Depth based foveated rendering for display systems

ABSTRACT

Methods and systems for depth-based foveated rendering in the display system are disclosed. The display system may be an augmented reality display system configured to provide virtual content on a plurality of depth planes using different wavefront divergence. Some embodiments include determining a fixation point of a user&#39;s eyes. Location information associated with a first virtual object to be presented to the user via a display device is obtained. A resolution-modifying parameter of the first virtual object is obtained. A particular resolution at which to render the first virtual object is identified based on the location information and the resolution-modifying parameter of the first virtual object. The particular resolution is based on a resolution distribution specifying resolutions for corresponding distances from the fixation point. The first virtual object rendered at the identified resolution is presented to the user via the display system.

PRIORITY CLAIM

This application is a continuation of U.S. application Ser. No.16/355,603, filed on Mar. 15, 2019, which claims the benefit of priorityof U.S. Provisional Application No. 62/644,366, filed on Mar. 16, 2018,the entirety of which is incorporated herein by reference.

INCORPORATION BY REFERENCE

This application incorporates by reference the entirety of each of thefollowing patent applications and publications: U.S. application Ser.No. 14/555,585 filed on Nov. 27, 2014, published on Jul. 23, 2015 asU.S. Publication No. 2015/0205126; U.S. application Ser. No. 14/690,401filed on Apr. 18, 2015, published on Oct. 22, 2015 as U.S. PublicationNo. 2015/0302652; U.S. application Ser. No. 14/212,961 filed on Mar. 14,2014, now U.S. Pat. No. 9,417,452 issued on Aug. 16, 2016; U.S.application Ser. No. 14/331,218 filed on Jul. 14, 2014, published onOct. 29, 2015 as U.S. Publication No. 2015/0309263; U.S. applicationSer. No. 15/902,927 filed on Feb. 22, 2018; U.S. Provisional ApplicationNo. 62/475,012 filed on Mar. 22, 2017; and U.S. Provisional ApplicationNo. 62/539,934 filed on Aug. 1, 2017.

BACKGROUND Field

The present disclosure relates to display systems, including augmentedreality imaging and visualization systems.

Description of the Related Art

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, in which digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves the presentation of digital or virtual imageinformation without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user. A mixed reality,or “MR”, scenario is a type of AR scenario and typically involvesvirtual objects that are integrated into, and responsive to, the naturalworld. For example, an MR scenario may include AR image content thatappears to be blocked by or is otherwise perceived to interact withobjects in the real world.

Referring to FIG. 1 , an augmented reality scene 10 is depicted. Theuser of an AR technology sees a real-world park-like setting 20featuring people, trees, buildings in the background, and a concreteplatform 30. The user also perceives that he/she “sees” “virtualcontent” such as a robot statue 40 standing upon the real-world platform30, and a flying cartoon-like avatar character 50 which seems to be apersonification of a bumble bee. These elements 50, 40 are “virtual” inthat they do not exist in the real world. Because the human visualperception system is complex, it is challenging to produce AR technologythat facilitates a comfortable, natural-feeling, rich presentation ofvirtual image elements amongst other virtual or real-world imageryelements.

Systems and methods disclosed herein address various challenges relatedto AR and VR technology.

SUMMARY

According to some embodiments, a system comprises one or more processorsand one or more computer storage media storing instructions that whenexecuted by the one or more processors, cause the one or more processorsto perform operations. The operations comprise monitoring, based oninformation detected via one or more sensors, eye movements of a user. Afixation point at which the user's eyes are fixating is determined basedon the eye movements, with the fixation point being a three-dimensionallocation in a field of view of the user. The operations includeobtaining location information associated with one or more virtualobjects to present to the user, the location information indicatingthree-dimensional positions of the virtual objects. The operations alsoinclude adjusting resolutions of at least one virtual object based, atleast in part, on a proximity of the at least one virtual object to thefixation point. The operations also include causing a presentation tothe user, via a display, of the virtual objects, with at least onevirtual object being rendered according to the adjusted resolution.

According to some embodiments, a display system comprises a displaydevice configured to present virtual content to a user, one or moreprocessors, and one or more computer storage media storing instructionsthat when executed by the system, cause the system to performoperations. The operations comprise monitoring information associatedwith eye movements of the user. A fixation point within a displayfrustum of the display device is determined based on the monitoredinformation, the fixation point indicating a three-dimensional locationbeing fixated upon by eyes of the user. The operations also includepresenting virtual content at three-dimensional locations within thedisplay frustum based on the determined fixation point, with the virtualcontent being adjusted in resolution based on a proximity of the virtualcontent from the fixation point.

According to some other embodiments, a method comprises monitoring,based on information detected via one or more sensors, eye movements ofa user. A fixation point at which the user's eyes are fixating isdetermined based on the eye movements, with the fixation point being athree-dimensional location in a field of view of the user. Locationinformation associated with one or more virtual objects to present tothe user is obtained, the location information indicatingthree-dimensional positions of the virtual objects. The resolution of atleast one virtual object is adjusted based, at least in part, on aproximity of the at least one virtual object to the fixation point. Themethod also includes causing presentation to the user, via a display, ofthe virtual objects, with at least one virtual object being renderedaccording to the adjusted resolution.

According to some embodiments, a display system comprises a frameconfigured to mount on a head of a user, a light modulating systemconfigured to output light to form images, and one or more waveguidesattached to the frame and configured to receive the light from the lightmodulating system and to output the light across a surface of the one ormore waveguides. The system also comprises one or more processors, andone or more computer storage media storing instructions that, whenexecuted by the one or more processors, cause the one or more processorsto perform various operations. The operations include determining anamount of light reaching a retina of an eye of the user; and adjustingresolution of virtual content to be presented to the user based on theamount of light reaching the retina.

According to some other embodiments, a display system comprises one ormore processors; and one or more computer storage media storinginstructions. When the instructions are executed by the one or moreprocessors, they cause the one or more processors to perform variousoperations. The operations include determining an amount of lightreaching a retina of an eye of a user of the display system; andadjusting resolution of virtual content to be presented to the userbased on the amount of light reaching the retina.

According to some embodiments, a method is performed by a display systemcomprising one or more processors and a head-mountable display. Themethod comprises determining an amount of light reaching a retina of aneye of a user of the display system; and adjusting resolution of virtualcontent to be presented to the user based on the amount of lightreaching the retina.

According to some other embodiments, a display system comprises a frameconfigured to mount on a head of a user; and light modulating system;one or more waveguides; one or more processors; and one or more computerstorage media storing instructions. The light modulating system isconfigured to output light to form images. The one or more waveguidesare attached to the frame and configured to receive the light from thelight modulating system and to output the light across a surface of theone or more waveguides. The one or more computer storage media storeinstructions that, when executed by the one or more processors, causethe one or more processors to perform various operations. The operationscomprise adjusting a resolution of component color images formingvirtual content based on: a proximity of the virtual content from a userfixation point; and a color of the component color image. At least oneof the component color images differs in resolution from component colorimages of another color.

According to yet other embodiments, a display system comprises one ormore processors; and one or more computer storage media storinginstructions. When the instructions are executed by the one or moreprocessors, they cause the one or more processors to perform variousoperations. The operations include adjusting a resolution of componentcolor images forming virtual content based on: a proximity of thevirtual content from a user fixation point; and a color of the componentcolor image, wherein at least one of the component color images differsin resolution from component color images of another color.

According to some other embodiments, a method is performed by a displaysystem comprising one or more processors and a head-mountable display.The method comprises adjusting a resolution of component color imagesforming virtual content based on: a proximity of the virtual contentfrom a user fixation point; and a color of the component color image,wherein at least one of the component color images differs in resolutionfrom component color images of another color.

According to yet other embodiments, a display system comprises an imagesource comprising a spatial light modulator for providing a first imagestream and a second image stream; a viewing assembly; one or moreprocessors in communication with the image source; and one or morecomputer storage media storing instructions that, when executed by theone or more processors, cause the one or more processors to performvarious operations. The viewing assembly comprises light guiding opticsfor receiving the first and second image streams from the image sourceand outputting the first and second image streams to a user. The variousoperations performed by the one or more processors comprise causing theimage source to output the first image stream to the viewing assembly,wherein images formed by the first image stream have a first pixeldensity; and causing the image source to output the second image streamto the viewing assembly. The images formed by the second image streamhave a second pixel density that is greater than the first pixeldensity, and correspond to portions of images provided by the firstimage stream. Images formed by the second image stream overliecorresponding portions of a field of view of provided by the first imagestream.

According to some embodiments, a wearable display system may include anafocal magnifier with circular polarization handedness dependentmagnification. The afocal magnifier may include a first fixed focallength lens element, a first geometric phase lens that exhibits apositive refractive power for a first handedness of incident circularlypolarized light and exhibits a negative refractive power for a secondhandedness of incident circularly polarized light, and a secondgeometric phase lens.

According to some other embodiments, an optical subsystem for a wearableimage projector may include a polarization selective reflector and a setof four lens elements positioned about the polarization selectivereflector.

According to some other embodiments, a display system for projectingimages to an eye of a user may include an eyepiece. The eyepiece mayinclude a waveguide, and an in-coupling grating optically coupled to thewaveguide. The display system may further include a first image sourceconfigured to project a first light beam associated with a first imagestream. The first image stream may have a first field of view and may beincident on a first surface of the in-coupling grating. A portion of thefirst light beam may be coupled into the waveguide by the in-couplinggrating for positioning the first image stream in a fixed position tothe eye of the user. The display system may further include a secondimage source configured to project a second light beam associated with asecond image stream. The second image stream may have a second field ofview that is narrower than the first field of view. The display systemmay further include a scanning mirror configured to receive and reflectthe second light beam such that the second light beam is incident on asecond surface of the in-coupling grating opposite to the first surfacethereof. A portion of the second light beam may be coupled into thewaveguide by the in-coupling grating. The display system may furtherinclude an eye-gaze tracker configured to detect movement of the eye ofthe user, and control circuitry in communication with the eye gazetracker and the scanning mirror. The control circuitry may be configuredto position the scanning mirror such that a position of the second imagestream is moved in accordance with the detected movement of the eye ofthe user.

According to some other embodiments, a display system for projectingimages to an eye of a user may include an eyepiece. The eyepiece mayinclude a waveguide, and an in-coupling grating optically coupled to thewaveguide. The display system may further include an image sourceconfigured to project a first light beam associated with a first imagestream in a first polarization, and a second light beam associated witha second image stream in a second polarization different from the firstpolarization. The first image stream may have a first field of view andthe second image stream may have a second field of view that is narrowerthan the first field of view. The first light beam and the second lightbeam may be multiplexed. The display system may further include apolarization beam splitter configured to receive and reflect the firstlight beam along a first optical path, and receive and transmit thesecond light beam along a second optical path. The display system mayfurther include a first optical reflector positioned along the firstoptical path and configured to receive and reflect the first light beamsuch that the first light beam is incident on a first surface of thein-coupling grating. A portion of the first light beam may be coupledinto the waveguide by the in-coupling grating for positioning the firstimage stream in a fixed position to the eye of the user. The displaysystem may further include a scanning mirror disposed along the secondoptical path and configured to receive and reflect the second lightbeam, and a second optical reflector positioned along the second opticalpath downstream from the scanning mirror. The second optical reflectormay be configured to receive and reflect the second light beam such thatthe second light beam is incident on a second surface of the in-couplinggrating opposite the first surface thereof. A portion of the secondlight beam may be coupled into the waveguide by the in-coupling grating.The display system may further include an eye-gaze tracker configured todetect movement of the eye of the user, and control circuitry incommunication with the eye gaze tracker and the scanning mirror. Thecontrol circuitry may be configured to position the scanning mirror suchthat a position of the second image stream is moved in accordance withthe detected movement of the eye of the user.

According to some other embodiments, a display system for projectingimages to an eye of a user may include a waveguide, an image sourceconfigured to project a first light beam associated with a first imagestream in a first polarization and a second light beam associated with asecond image stream in a second polarization different from the firstpolarization. The first image stream may have a first field of view, andthe second image stream having a second field of view that is narrowerthan the first field of view. The first light beam and the second lightbeam may be multiplexed. The display system may further include apolarization beam splitter configured to receive and reflect the firstlight beam along a first optical path, and to receive and transmit thesecond light beam along a second optical path. The display system mayfurther include a first in-coupling prism positioned along the firstoptical path and adjacent a first surface of the waveguide. The firstin-coupling prism may be configured to couple a portion of the firstlight beam into the waveguide for positioning the first image stream ina fixed position to the eye of the user. The display system may furtherinclude a scanning mirror disposed along the second optical path andconfigured to receive and reflect the second light beam. The displaysystem may further include a second in-coupling prism positioned alongthe second optical path downstream from the scanning mirror and adjacenta second surface of the waveguide opposite to the first surface of thewaveguide. The second in-coupling prism may be configured to couple aportion of the second light beam into the waveguide. The display systemmay further include an eye-gaze tracker configured to detect movement ofthe eye of the user, and control circuitry in communication with the eyegaze tracker and the scanning mirror. The control circuitry may beconfigured to position the scanning mirror such that a position of thesecond image stream is moved in accordance with the detected movement ofthe eye of the user

According to an embodiment, a display system for projecting images to aneye of a user includes an image source. The image source can beconfigured to project a first light beam associated with a first imagestream in a first polarization, and a second light beam associated witha second image stream in a second polarization different from the firstpolarization. The first image stream can have a first field of view, andthe second image stream can have a second field of view that is narrowerthan the first field of view. The first light beam and the second lightbeam can be multiplexed. The display system can further include apolarization beam splitter. The polarization beam splitter can beconfigured to receive and reflect the first light beam along a firstoptical path toward a viewing assembly for positioning the first imagestream in a fixed position to the eye of the user, and receive andtransmit the second light beam along a second optical path. The displaysystem can further include a scanning mirror disposed along the secondoptical path and configured to receive and reflect the second light beamtoward the viewing assembly. The display system can further include aneye-gaze tracker configured to detect movement of the eye of the user,and control circuitry in communication with the eye gaze tracker and thescanning mirror. The control circuitry can be configured to position thescanning mirror such that a position of the second image stream is movedin accordance with the detected movement of the eye of the user.

According to another embodiment, a display system for projecting imagesto an eye of a user include an image source. The image source can beconfigured to project a first light beam associated with a first imagestream and a second light beam associated with a second image stream.The first image stream can have a first field of view, and the secondimage stream can have a second field of view that is narrower than thefirst field of view. The first light beam and the second light beam canbe multiplexed. The display system can further include a scanning mirrorconfigured to receive and reflect the first light beam and the secondlight beam toward a viewing assembly for projecting the first imagestream and the second image stream. The display system can furtherinclude an eye-gaze tracker configured to detect movement of the eye ofthe user, and control circuitry in communication with the eye gazetracker and the scanning mirror. The control circuitry can be configuredto position the scanning mirror such that a position of the first imagestream and a position of the second image stream are moved in accordancewith the detected movement of the eye of the user. The display systemcan further include a switchable optical element disposed in an opticalpath of the first light beam and the second light beam. The switchableoptical element can be configured to be switched to a first state forthe first light beam such that the first light beam is angularlymagnified by a first angular magnification, and be switched to a secondstate for the second light beam such that the second light beam isangularly amplified by a second angular magnification that is less thanthe first angular magnification.

In some embodiments, a display system comprises one or more processorsand one or more computer storage media storing instructions that whenexecuted by the one or more processors, cause the one or more processorsto perform operations. The operations comprise determining a fixationpoint of a user's eyes; obtaining location information associated with afirst virtual object to be presented to the user via a display device;obtaining a resolution-modifying parameter of the first virtual object;identifying, based on the location information and theresolution-modifying parameter of the first virtual object, a particularresolution at which to render the first virtual object, wherein theparticular resolution is based on a resolution distribution specifyingresolutions for corresponding distances from the fixation point; andcausing presentation to the user, via the display device, of the firstvirtual object rendered at the identified resolution.

In some embodiments, a computer-implemented method is provided. Themethod is performed by a display system of one or more processors. Themethod comprises determining a fixation point of a user's eyes;obtaining location information associated with a first virtual object tobe presented to the user via a display device; obtaining aresolution-modifying parameter of the first virtual object; identifying,based on the location information and the resolution-modifying parameterof the first virtual object, a particular resolution at which to renderthe first virtual object, wherein the particular resolution is based ona resolution distribution specifying resolutions for correspondingdistances from the fixation point; and causing presentation to the user,via the display device, of the first virtual object rendered at theidentified resolution.

In some embodiments, a non-transitory computer storage media isprovided. The computer storage media stores instructions that whenexecuted by a display system of one or more processors, cause the one ormore processors to perform operations. The operations comprisedetermining a fixation point of a user's eyes; obtaining locationinformation associated with a first virtual object to be presented tothe user via a display device; obtaining a resolution-modifyingparameter of the first virtual object; identifying, based on thelocation information and the resolution-modifying parameter of the firstvirtual object, a particular resolution at which to render the firstvirtual object, wherein the particular resolution is based on aresolution distribution specifying resolutions for correspondingdistances from the fixation point; and causing presentation to the user,via the display device, of the first virtual object rendered at theidentified resolution.

Additional examples of embodiments are provided below.

1. A display system comprising:

-   -   one or more processors; and    -   one or more computer storage media storing instructions that        when executed by the one or more processors, cause the one or        more processors to perform operations comprising:        -   determining a fixation point of a user's eyes;        -   obtaining location information associated with a first            virtual object to be presented to the user via a display            device;        -   obtaining a resolution-modifying parameter of the first            virtual object;        -   identifying, based on the location information and the            resolution-modifying parameter of the first virtual object,            a particular resolution at which to render the first virtual            object, wherein the particular resolution is based on a            resolution distribution specifying resolutions for            corresponding distances from the fixation point; and        -   causing presentation to the user, via the display device, of            the first virtual object rendered at the identified            resolution.

2. The display system of example 1, wherein the resolution-modifyingparameter comprises a content type associated with the first virtualobject, wherein the operations further comprise:

-   -   accessing a plurality of resolution distributions, the        resolution distributions associated with respective virtual        content types; and    -   selecting, from the plurality of resolution distributions, a        particular resolution distribution based on the content type of        the first virtual object, wherein the particular resolution        comprises the particular resolution distribution.

3. The display system of example 3, wherein the virtual content typeassociated with the first virtual object is identified based on afrequency spectrum associated with the first virtual object.

4. The display system of example 3, wherein the plurality of resolutiondistributions are associated with respective rollofs in resolution awayfrom the fixation point, wherein values of the rollofs differ forcontent with different frequency spectrums.

5. The display system of example 1, wherein the resolution-modifyingparameter is a user selectable value.

6. The display system of example 5, wherein the display device isconfigured to adjust the particular resolution, and wherein adjustingthe particular resolution comprises:

-   -   causing presentation to the user, via the display device, of a        second virtual object, the second virtual object being rendered        with the resolution distribution identified for the first        virtual object;    -   receiving, from the user, a response indicating user detection        of reduction in resolution of the second virtual object, wherein        the user response is the user selectable value; and    -   adjusting the particular resolution distribution.

7. The display system of example 6, wherein adjusting the particularresolution distribution comprises:

-   -   adjusting a rolloff associated with the particular resolution        distribution, wherein adjusting a rolloff changes an amount of        resolution reduction based on angular distance from the center        of the field of view of the user.

8. The display system of example 1, wherein the fixation point is in avolume at the center of the field of view of the user.

9. The display system of example 1, wherein the field of view of theuser is separated into a plurality of portions based on the resolutiondistribution, the plurality of portions including the first portion,wherein each portion encompasses a respective range of angular distancesfrom the center of the field of view, and wherein each portion isassigned an associated resolution at which to render virtual content.

10. The display system of example 9, wherein the operations furthercomprise:

-   -   determining a proximity of the first virtual object to a        boundary of one of the plurality of portions; and    -   modifying a presentation of the first virtual objected based on        the determined proximity.

11. The display system of example 9, wherein modifying a presentation ofthe first virtual objected based on the determined proximity comprisesapplying a blurring process to the virtual object.

12. The display system of example 9, wherein identifying a particularresolution at which to render the first virtual object comprises:

-   -   identifying a second portion of the plurality of portions which        encompasses the first virtual object; and    -   identifying the resolution based on the second portion.

13. A computer-implemented method, the method being performed by adisplay system of one or more processors, and the method comprising:

-   -   determining a fixation point of a user's eyes;    -   obtaining location information associated with a first virtual        object to be presented to the user via a display device;    -   obtaining a resolution-modifying parameter of the first virtual        object;    -   identifying, based on the location information and the        resolution-modifying parameter of the first virtual object, a        particular resolution at which to render the first virtual        object, wherein the particular resolution is based on a        resolution distribution specifying resolutions for corresponding        distances from the fixation point; and    -   causing presentation to the user, via the display device, of the        first virtual object rendered at the identified resolution.

14. The computer-implemented method of example 13, wherein theresolution-modifying parameter comprises a content type associated withthe first virtual object, wherein the method further comprises:

-   -   accessing a plurality of resolution distributions, the        resolution distributions associated with respective virtual        content types; and    -   selecting, from the plurality of resolution distributions, a        particular resolution distribution based on the content type of        the first virtual object, wherein the particular resolution        comprises the particular resolution distribution.

15. The computer-implemented method of example 14, wherein the virtualcontent type associated with the first virtual object is identifiedbased on a frequency spectrum associated with the first virtual object.

16. The computer-implemented method of example 14, wherein the pluralityof resolution distributions are associated with respective rolloffs inresolution away from the fixation point, wherein values of the rolloffsdiffer for content with different frequency spectrums.

17. Non-transitory computer storage media storing instructions that whenexecuted by a display system of one or more processors, cause the one ormore processors to perform operations comprising:

-   -   determining a fixation point of a user's eyes;    -   obtaining location information associated with a first virtual        object to be presented to the user via a display device;    -   obtaining a resolution-modifying parameter of the first virtual        object;    -   identifying, based on the location information and the        resolution-modifying parameter of the first virtual object, a        particular resolution at which to render the first virtual        object, wherein the particular resolution is based on a        resolution distribution specifying resolutions for corresponding        distances from the fixation point; and    -   causing presentation to the user, via the display device, of the        first virtual object rendered at the identified resolution.

18. The computer storage media of example 17, wherein theresolution-modifying parameter comprises a content type associated withthe first virtual object, wherein the operations further comprise:

-   -   accessing a plurality of resolution distributions, the        resolution distributions associated with respective virtual        content types; and    -   selecting, from the plurality of resolution distributions, a        particular resolution distribution based on the content type of        the first virtual object, wherein the particular resolution        comprises the particular resolution distribution.

19. The computer storage media of example 18, wherein the virtualcontent type associated with the first virtual object is identifiedbased on a frequency spectrum associated with the first virtual object.

20. The computer storage media of example 18, wherein the plurality ofresolution distributions are associated with respective rolloffs inresolution away from the fixation point, wherein values of the rolloffsdiffer for content with different frequency spectrums.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user's view of augmented reality (AR) through an ARdevice.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIGS. 3A-3C illustrate relationships between radius of curvature andfocal radius.

FIG. 4A illustrates a representation of the accommodation-vergenceresponse of the human visual system.

FIG. 4B illustrates examples of different accommodative states andvergence states of a pair of eyes of the user.

FIG. 4C illustrates an example of a representation of a top-down view ofa user viewing content via a display system.

FIG. 4D illustrates another example of a representation of a top-downview of a user viewing content via a display system.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user.

FIG. 7 illustrates an example of exit beams outputted by a waveguide.

FIG. 8 illustrates an example of a stacked waveguide assembly in whicheach depth plane includes images formed using multiple differentcomponent colors.

FIG. 9A illustrates a cross-sectional side view of an example of a setof stacked waveguides that each includes an incoupling optical element.

FIG. 9B illustrates a perspective view of an example of the plurality ofstacked waveguides of FIG. 9A.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B.

FIG. 9D illustrates an example of wearable display system.

FIG. 10A illustrates an example of a representation of a top-down viewof a user viewing content via a display system.

FIG. 10B illustrates another example of a representation of a top-downview of a user viewing content via a display system.

FIG. 10C illustrates yet another example of a representation of atop-down view of a user viewing content via a display system.

FIG. 10D is a block diagram of an example display system.

FIG. 11A1 illustrates an example of a representation of a top-down viewof adjustments in resolution in different resolution adjustment zonesbased on three-dimensional fixation point tracking.

FIG. 11A2 illustrates examples of representations of top-down views ofresolution adjustment zones at different times as the sizes and numbersof the zones change.

FIG. 11B illustrates an example of a three-dimensional representation ofa portion of the resolution adjustment zones of FIG. 11A1.

FIG. 11C illustrates another example of a configuration for resolutionadjustment zones.

FIG. 11D illustrates an example of a three-dimensional representation ofthe resolution adjustment zones of FIG. 11C.

FIG. 11E illustrates another example of a three-dimensionalrepresentation of the resolution adjustment zones of FIG. 11C.

FIGS. 12A-12C shown diagrams of examples of processes for adjustingresolutions of content according to proximity to a three-dimensionalfixation point.

FIG. 13 illustrates an example of a representation of a user viewingmultiple virtual objects aligned with the user's line of sight.

FIG. 14 is a diagram of an example of a process for adjusting virtualcontent based on angular proximity to a user's gaze.

FIG. 15 illustrates an example of a representation of the retina of aneye of a user.

FIG. 16 graphically illustrates an example of resolution, and rod andcone density, across the retina of FIG. 15 .

FIG. 17 graphically illustrates an example of the relationship betweenpupil size and the amount of light incident on an eye of a user.

FIG. 18 is a diagram of an example of a process for adjusting virtualcontent based on the amount of light incident on an eye of a user.

FIG. 19 graphically illustrates an example of a change in resolutiondetectable by the eye of a user as the amount of light incident on theeye changes.

FIG. 20 graphically illustrates an example of differences in sensitivityof the eye to light of different colors at different levels ofillumination.

FIG. 21 is a diagram of an example of a process for adjusting virtualcontent formed using multiple component color images, where theresolution adjustment is made based on the color of the component colorimage.

FIGS. 22A-22C illustrate examples of changing contrast sensitivity asthe amount of light incident on the eye of the user decreases.

FIG. 23 illustrates an example of a representation of the optic nerveand peripheral blind spots of the eyes of a user.

FIG. 24 shows an exemplary monocular field of view for a human eye.

FIG. 25A shows an exemplary wearable display device configured toprovide virtual content to a user.

FIG. 25B is a block diagram depicting an augmented reality system.

FIG. 25C illustrates schematically light paths in a viewing opticsassembly (VOA) that may be used to present a digital or virtual image toa viewer.

FIGS. 26A-26D illustrate exemplary render perspectives to be used andlight fields to be produced in an AR system for each of two exemplaryeye orientations.

FIGS. 26E-26F illustrate schematically an exemplary configuration ofimages that can be presented to a user.

FIGS. 26G-26H illustrate schematically exemplary configurations ofimages that can be presented to a user.

FIG. 27 illustrates a field of view and a field of regard as shown inFIG. 24 , overlaid upon one of the displays in the wearable displaydevice as shown in FIG. 25 .

FIGS. 28A-28B illustrate some of the principles described in FIGS.26A-26D.

FIGS. 28C-28D illustrate some exemplary images that can be presented toa user.

FIG. 28E illustrates an exemplary high-FOV low-resolution image frame.

FIG. 28F illustrates an exemplary low-FOV high-resolution image frame.

FIG. 29A shows a simplified block diagram of a display system.

FIG. 29B illustrates schematically a cross-sectional view of anaugmented reality (AR) system.

FIGS. 30A-30B illustrate schematically a display system for projectingimage streams to an eye of a user.

FIG. 30C illustrates schematically a cross-sectional view of anaugmented reality (AR) system.

FIG. 30D shows a simplified block diagram of a display system.

FIG. 31A illustrates schematically the operating principles of a firstrelay lens assembly in the display system illustrated in FIGS. 30A-30B.

FIG. 31B illustrates schematically the operating principles of a secondrelay lens assembly in the display system illustrated in FIGS. 30A-30B.

FIGS. 31C-31D illustrate schematically a display system.

FIGS. 32A-32C illustrate schematically a display system.

FIGS. 33A-33B illustrate schematically a display system.

FIGS. 34A-34B illustrate schematically a display system.

FIG. 35 illustrates schematically a display system.

FIG. 36 illustrates schematically an augmented reality near-eye displaysystem.

FIG. 37A is a schematic illustration of a dual magnification afocalmagnifier.

FIG. 37B is a schematic illustration of a dual focal magnificationafocal magnifier.

FIGS. 38A-38B illustrates schematically an exemplary configuration ofimages that can be presented to a user.

FIGS. 39A-39B illustrate some exemplary images that can be presented toa user.

FIGS. 40A-40D illustrate schematically a display system for projectingimage streams to an eye of a user.

FIGS. 41A-41D illustrate schematically a display system for projectingimage streams to an eye of a user.

FIG. 42 illustrate an exemplary frame structure for a high-FOVlow-resolution image stream and a low-FOV high-resolution image streamthat are time-division multiplexed.

FIG. 43 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 44 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 45 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 46 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 47 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 48 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 49 illustrates schematically a display system for projecting imagestreams to an eye of a user.

FIG. 50 illustrates schematically a display system for projecting imagestreams to an eye of a user according to some embodiments.

FIG. 51 illustrates schematically a display system for projecting imagestreams to an eye of a user according to some embodiments.

FIGS. 52A-52B illustrate schematically a display system for projectingimage streams to an eye of a user according to some embodiments.

FIGS. 53A-53B illustrate schematically a display system for projectingimage streams to an eye of a user according to some embodiments.

FIG. 54 illustrates a representation of an angular field of view of auser along with an example resolution distribution.

FIG. 55A-55B illustrate example schemes to identify a rolloff for aresolution distribution based on a type of virtual content.

FIG. 55C-55D illustrate graphs of average rolloffs determined fordifferent types of image content.

FIG. 56 illustrates a flowchart of an example process 5600 fordetermining a rolloff to be utilized in a resolution distribution.

FIG. 57 illustrates an example flowchart of a process for presentingvirtual content according to a type of the virtual content.

FIG. 58A illustrates two example blurring regions.

FIG. 58B illustrates two additional example blurring regions.

FIG. 59 illustrates an example of different resolution adjustment zonesaccording to the techniques described herein.

DETAILED DESCRIPTION

Rendering virtual content for augmented and virtual display systems iscomputationally intensive. Among other things, the computationalintensity may undesirably use large amounts of memory, cause highlatency, and/or may require the use of powerful processing units thatmay have high cost and/or high energy-consumption.

In some embodiments, methods and systems conserve computationalresources, such as memory and processing time, by reducing theresolution of virtual content positioned at locations away from thefixation point of the user's eyes. For example, the system may rendervirtual content at a relative high (e.g., a highest) resolution at orproximate a fixation point of the user's eyes, while utilizing one ormore lower resolutions for virtual content away from the fixation point.The virtual content is presented by a display system that can displayvirtual content on a plurality of different depths (e.g., a plurality ofdifferent depth planes, such as two or more depth planes), and thereduction in resolution preferably occurs along at least the z axis,where the z-axis is the depth axis (corresponding to distance away fromthe user). In some embodiments, the resolution reduction occurs alongthe z-axis and one or both of the x and y axes, where the x-axis is thelateral axis, and the y-axis is the vertical axis.

Determining the appropriate resolution of the virtual content mayinclude determining the fixation point, in three-dimensional space, of auser's eyes. For example, the fixation point may be an x, y, z,coordinate in a field of view of the user, upon which the user's eyesare fixated. The display system may be configured to present virtualobjects that have differences in resolution, with the resolutiondecreasing with decreasing proximity of a virtual object to the fixationpoint; stated another way, the resolution decreases with increasingdistance from the fixation point.

As discussed herein, the display system may present virtual objectswithin a display frustum of the display system, with the virtual objectscapable of being presented on different depth planes. In someembodiments, the display frustum is the field of view provided by thedisplay system, over which the display system is configured to presentvirtual content to the user of the display system. The display systemmay be a head-mounted display system including one or more waveguideswhich may present virtual content (e.g., virtual objects, graphics,text, and so on), with the one or more waveguides configured to outputlight with different wavefront divergence and/or different binoculardisparity corresponding to the different depth planes (e.g.,corresponding to particular distances from the user). It will beappreciated that each eye may have an associated one or more waveguides.Using the different wavefront divergence and/or different binoculardisparity, the display system may cause a first virtual object to appearto be located at a first depth in the user's field of view, whilecausing a second virtual object to appear to be located at a seconddepth in the user's field of view. In some embodiments, the depth planeof or a close depth plane to the fixation point may be determined andthe resolution of content on other depth planes may be reduced based ondistance of those depth planes to the depth plane on which the fixationpoint is disposed. It will be appreciated that references to the depthof virtual content herein (the distance of the virtual content from theuser on the z-axis) refer to the apparent depth of the virtual contentas intended to be seen to the user; in some embodiments, the depth ofthe virtual object may be understood to be the distance from the user ofa real object having wavefront divergence and/or binocular disparitysimilar to that of the virtual object.

It will be appreciated that the proximity of a virtual object to thefixation point may be determined by various measures, non-limitingexamples of which include determining the distance between the fixationpoint and the virtual object, determining the resolution adjustment zoneoccupied by the virtual object relative to a resolution adjustment zoneoccupied by the fixation point (in embodiments where the user's field ofview is subdivided into resolution adjustment zones as described below),and determining the angular proximity of the virtual object to thefixation point of the user. The proximity may also be determined using acombination of the above-noted techniques. For example, the distanceand/or angular proximity of a first zone (in which a virtual object islocated) to a second zone (in which the fixation point is located) maybe used to determine proximity. These various measures are furtherdiscussed below.

In some embodiments, determining the fixation point may includeanticipating the fixation point of the user's eyes and utilizing theanticipated fixation point as the fixation point for determining theresolution of virtual content. For example, the display system mayrender particular content at a relatively high resolution with theexpectation that the user's eyes will fixate on that content. As anexample, it will be appreciated that the human visual system may besensitive to sudden changes in a scene (e.g., sudden motion, changes inluminance, etc.). In some embodiments, the display system may determinethat the virtual content is of a type (e.g., involving motion in a scenein which other virtual and real objects are still) that would cause theuser's eyes to fixate on it, and then render that virtual content athigh resolution with the expectation that the user's eyes willsubsequently focus on that virtual content.

As noted above, in some embodiments, the distance from the determinedfixation point to a virtual object may correspond to a distanceextending in three-dimensions. As an example, a first virtual objectlocated on a same depth from the user (e.g., at the same depth plane) asthe determined fixation point, but located horizontally orlongitudinally from the fixation point, may be similarly reduced inresolution as a second virtual object located at a further depth (e.g.,a further depth plane) from the determined fixation point. Consequently,different resolutions may be associated with different distances fromthe fixation point.

In some embodiments, the environment around the user may be broken intovolumes of space (herein also referred to as resolution adjustmentzones) with the resolution of virtual objects in the same resolutionadjustment zone being similar. The resolution adjustment zones may havearbitrary three-dimensional shapes, e.g., cubes, or otherthree-dimensional polygonal shapes, or curved three-dimensional shapes,as described herein. In some embodiments, all resolution adjustmentzones have similar shapes, e.g., cuboid or spherical. In some otherembodiments, different resolution adjustment zones may have differentshapes or sizes (e.g., the shapes and/or sizes of the volumes may changewith distance from the fixation point).

In some embodiments, the resolution adjustment zones are portions of theuser's field of view. For instance, the field of view of the user may beseparated into volumes of space forming the resolution adjustment zones.In some embodiments, each depth plane may be subdivided into one or morecontiguous volumes of space, that is, one or more resolution adjustmentzones. In some embodiments, each resolution adjustment zone canencompass a particular range of depths from the user (e.g., a depthplane value+/−a variance, wherein examples of variances include 0.66dpt, 0.50 dpt, 0.33 dpt, or 0.25 dpt), and a particular lateral and aparticular vertical distance. Virtual objects located within the sameresolution adjustment zone as the determined fixation point may bepresented (e.g., rendered) at a high (e.g., full) resolution, whilevirtual objects located in volumes of space outside of the fixationpoint's resolution adjustment zone may be rendered at lesser resolutionsaccording to a distance of the volumes from the fixation point's volumeof space. In some embodiments, each resolution adjustment zone may beassigned a particular resolution (e.g., a particular reduction inresolution relative to the full resolution) and virtual content fallingwithin a given zone may be rendered at the associated resolution forthat zone. In some embodiments, the distance between a volume and thevolume occupied by the fixation point may be determined, and theresolution may be set based upon this distance.

Advantageously, the number and sizes of the resolution adjustment zonesutilized to break up a user's field of view may be modified according toa confidence in the user's determined fixation point. For example, thesize associated with each volume of space may be increased or decreasedbased on the confidence that the user's gaze is verging on a precisepoint in three-dimensional space. If a confidence in the fixation pointis high, the display system may present only virtual objects within acompact resolution adjustment zone at a relative high resolution (thecompact resolution adjustment zone including the fixation point), whilereducing resolutions of other virtual objects, and thus conservingprocessing power. However, if the confidence is low, the display systemmay increase the size of each volume of space (e.g., reduce an overallnumber of the volumes), such that each volume of space encompasses agreater number of virtual objects in the fixation point's volume ofspace. It will be appreciated that the sizes and shapes of the volumesmay be fixed during production of the display system, e.g., based uponexpected tolerances in systems for determining the fixation point,and/or may be adjusted or set in the field depending upon a user'scharacteristics, the user's environment, and/or changes in software thatchange the tolerances for the systems for determining the fixationpoint.

It will be appreciated that the user's sensitivity to resolution maydecrease with distance from the fixation point. Consequently, byensuring that full resolution content is presented at the fixation pointand by allowing a margin of error for where the fixation point islocated, the perceptibility of reductions in resolution may be reducedor eliminated, thereby providing the perception of a high-resolutiondisplay without utilizing the computational resources typically requiredto present content for such a high resolution display.

In some embodiments, the proximity of a virtual object to the fixationpoint may be determined based on an angular proximity of the virtualobject to a gaze of the user, and a resolution of the virtual object maydecrease as the angular proximity decreases. In some embodiments, thismay result in virtual objects located at different depths from the userbeing presented at a similar resolution. For example, a first virtualobject at a location corresponding to a user's determined fixation pointmay be located in front (e.g., closer in depth to the user) of a secondvirtual object. Since the second virtual object will be along a gaze ofthe user, and thus similarly fall on the user's fovea, where the user'seye is most sensitive to changes in resolution, the second virtualobject may optionally be presented at a similar (e.g. same) resolutionas the first virtual object. Optionally, the second virtual object maybe reduced in resolution, and further adjusted via a blurring process(e.g., a Gaussian blurring kernel may be convolved with the secondvirtual object), which may represent that the second virtual object isfurther (e.g., located on a farther depth plane) from the user.

The reductions in resolution may vary based upon how virtual content ispresented by the display systems. In some embodiments, a first exampledisplay system referred to herein as a vari-focal display system maypresent virtual content on different depth planes, with all content(e.g., virtual objects) presented at a same depth plane (e.g., via asame waveguide) at a time, e.g. for each frame presented to the user.That is, the vari-focal display system may utilize a single depth plane(e.g., selected from multiple depth planes based on a fixation point ofthe user, or selected based on a depth of a particular presented virtualobject) at a time to present content, and may change the depth plane insubsequent frames (e.g., select different depth planes). In some otherembodiments, a second example display system referred to herein as amulti-focal display system may present virtual content on differentdepth planes, with content simultaneously displayed on multiple depthplanes. As will be further described herein, the vari-focal displaysystem may optionally utilize a single frame buffer, and with respect tothe example above regarding blurring a second virtual object, the secondvirtual object may be blurred prior to presentation to the user from thesingle frame buffer. In contrast, the multi-focal display system maypresent the second virtual object on a further depth (e.g., on a furtherdepth plane) from the first virtual object optionally at a reducedresolution, and the second virtual object may appear to the user asbeing blurred (e.g., the second virtual object will be blurred based onthe natural physics of the user's eyes, without further processing).

As disclosed herein, the display system may present virtual objects atrelatively high (e.g. full) resolution at or near the determinedfixation point, and may present virtual objects at reduced resolutionsfarther from the fixation point. Preferably, the relatively highresolution is the highest resolution for presentation of virtual objectsin the user's field of view. The relatively high resolution may be amaximum resolution of the display system, a user-selectable resolution,a resolution based on specific computing hardware presenting the virtualobjects, and so on.

It will be appreciated that adjusting resolution of a virtual object mayinclude any modification to the virtual object to alter a quality ofpresentation of the virtual object. Such modifications may include oneor more of adjusting a polygon count of the virtual object, adjustingprimitives utilized to generate the virtual object (e.g., adjusting ashape of the primitives, for example adjusting primitives from trianglemesh to quadrilateral mesh, and so on), adjusting operations performedon the virtual object (e.g., shader operations), adjusting textureinformation, adjusting color resolution or depth, adjusting a number ofrendering cycles or a frame rate, and so on, including adjusting qualityat one or more points within a graphics pipeline of graphics processingunits (GPUs).

In some embodiments, on the x and y-axes, changes in the resolution ofvirtual content away from the fixation point may generally track changesin the distribution of photoreceptors in the retina of an eye of theuser. For example, it will be appreciated that a view of the world andof virtual content may be imaged on the retina, such that differentparts of the retina may be mapped to different parts of the user's fieldof view. Advantageously, the resolution of virtual content across theuser's field of view may generally track the density of correspondingphotoreceptors (rods or cones) across the retina. In some embodiments,the resolution reduction away from the fixation point may generallytrack the reduction in density of cones across the retina. In some otherembodiments, the resolution reduction away from the fixation point maygenerally track the reduction in density of rods across the retina. Insome embodiments, the trend of the resolution reduction away from thefixation point may be within ±50%, ±30%, ±20%, or ±10% of the trend inthe reduction in the density of rods and/or cones across the retina.

The rods and cones are active at different levels of incident light. Forexample, cones are active under relatively bright conditions, while rodsare active under relatively low light conditions. Consequently, in someembodiments where the reduction in resolution generally tracks thedensities of rods or cones across the retina, the display system may beconfigured to determine the amount of light incident on the retina.Based on this amount of light, the appropriate adjustment in resolutionmay be made. For example, the reduction in resolution may generallytrack the changes in the density of rods across the retina in low lightconditions, while the reduction in resolution may generally track thechanges in the density of cones in bright conditions. Consequently, insome embodiments, the display system may be configured to change theprofile of the reduction in image resolution based upon the amount oflight incident on the retina.

It will be appreciated that the ability of the human eye to resolve finedetails may not be directly proportional to the densities of rods orcones in the retina. In some embodiments, changes in the resolution ofvirtual content across the user's field of view generally track changesin the ability of the eye to resolve fine details. As noted above, theprogression of the changes in resolution of the virtual content may varywith the amount of light reaching the retina.

In some embodiments, the amount of light reaching the retina may bedetermined by detecting the amount of ambient light incident on a sensormounted on the display device. In some embodiments, determining theamount of light reaching the retina may also include determining theamount of light outputted by the display device to the user. In yetother embodiments, the amount of light reaching the retina may bedetermined by imaging the eye of the user to determine pupil size.Because pupil size is related to the amount of light reaching theretina, determining pupil size allows the amount of light reaching theretina to be extrapolated.

It will be appreciated that full color virtual content may be formed bya plurality of component color images, which, in the aggregate, providethe perception of full color. The human eye may have differentsensitivities to different wavelengths, or colors, of light. In someembodiments, in addition to changing based on proximity to a fixationpoint, the changes in resolution of the virtual content may vary basedupon the color of the component color image that is presented by thedisplay system. For example, were the component color images comprisered, green, and blue images, the green component color images may have ahigher resolution than the red component color images, which may have ahigher resolution than the blue component color images. In someembodiments, to account for changes in the sensitivities of the eye todifferent colors at different levels of incident light, the amount oflight reaching the retina may be determined, and the resolutionadjustment for a given component color image may also vary based uponthe determination of the amount of light reaching the retina.

It will be appreciated that the contrast sensitivity of the eye may alsovary based on the amount of light incident on the retina. In someembodiments, the size or total number of gradations in contrast in thevirtual content may vary based upon the amount of light reaching theretina. In some embodiments, the contrast ratio of images forming thevirtual content may vary based upon the amount of light incident on theretina, with the contrast ratio decreasing with decreasing amounts oflight.

In some embodiments, certain parts of the user's field of view may notbe provided with any virtual content. For example, the display systemmay be configured to not provide virtual content in a blind spot causedby the optic nerve and/or a peripheral blind spot of a given eye.

As discussed herein, the display system may be configured to displayhigh resolution content in one part of the user's field of view andlower resolution content in another part of the user's field of view. Itwill be appreciated that the high resolution content may have a higherpixel density than the lower resolution content. In some environments,the display system may be configured to provide such high and lowresolution content by effectively superimposing high-resolution and lowresolution images. For example, the system may display a low resolutionimage that spans the entire field of view, and then display a highresolution image spanning a small portion of the field of view, with thehigh-resolution image being located at the same location as acorresponding portion of the low resolution image. The high and lowresolution images may be routed through different optics, which outputlight at appropriate angles to determine how much of the field of viewthose images occupy.

In some embodiments, a single spatial light modulator (SLM) may be usedto encode light with image information, and a beam splitter or opticalswitch may be used to split a single light stream from the SLM into twostreams, one stream to propagate through optics for the low-resolutionimages and a second stream to propagate through optics for thehigh-resolution images. In some other embodiments, the polarization ofthe light encoded with image information may be selectively switched andpassed through optics that effectively provide different angularmagnifications for light of different polarizations, thereby providingthe high and low resolution images.

Advantageously, various embodiments disclosed herein reduce requirementsfor processing power for providing content on display systems. Since alarger share of processing power may be devoted to virtual objects thatare proximate to a user's three-dimensional fixation point, whileprocessing power for virtual objects further away may be reduced, theoverall required processing power for the display system may be reduced,thus reducing one or more of the size of processing components, the heatgenerated by the processing components, and the energy requirements forthe display system (e.g., the display system may optionally be batterypowered, require lower capacity batteries, and/or operate for a longerduration with a given battery). Therefore, embodiments described hereinaddress technological problems arising out of augmented or virtualreality display systems. Additionally, the described techniquesmanipulate graphical content such that upon presentation to the user,the graphical content is presented fundamentally differently (e.g.,resolutions are modified), while the graphical content may appear to theuser as being the same. Thus, the display system transforms graphicalcontent while preserving visual fidelity, and conserving processingpower, as the user looks around their ambient environment.

It will be appreciated that the display system may be part of anaugmented reality display system, or a virtual reality display system.As one example, the display of the display system may be transmissiveand may allow the user a view of the real world, while providing virtualcontent in the form of images, video, interactivity, and so on, to theuser. As another example, the display system may block the user's viewof the real world, and virtual reality images, video, interactivity, andso on, may be presented to the user.

Reference will now be made to the drawings, in which like referencenumerals refer to like parts throughout.

FIG. 2 illustrates a conventional display system for simulatingthree-dimensional imagery for a user. It will be appreciated that auser's eyes are spaced apart and that, when looking at a real object inspace, each eye will have a slightly different view of the object andmay form an image of the object at different locations on the retina ofeach eye. This may be referred to as binocular disparity and may beutilized by the human visual system to provide a perception of depth.Conventional display systems simulate binocular disparity by presentingtwo distinct images 190, 200 with slightly different views of the samevirtual object—one for each eye 210, 220—corresponding to the views ofthe virtual object that would be seen by each eye were the virtualobject a real object at a desired depth. These images provide binocularcues that the user's visual system may interpret to derive a perceptionof depth.

With continued reference to FIG. 2 , the images 190, 200 are spaced fromthe eyes 210, 220 by a distance 230 on a z-axis. The z-axis is parallelto the optical axis of the viewer with their eyes fixated on an objectat optical infinity directly ahead of the viewer. The images 190, 200are flat and at a fixed distance from the eyes 210, 220. Based on theslightly different views of a virtual object in the images presented tothe eyes 210, 220, respectively, the eyes may naturally rotate such thatan image of the object falls on corresponding points on the retinas ofeach of the eyes, to maintain single binocular vision. This rotation maycause the lines of sight of each of the eyes 210, 220 to converge onto apoint in space at which the virtual object is perceived to be present.As a result, providing three-dimensional imagery conventionally involvesproviding binocular cues that may manipulate the vergence of the user'seyes 210, 220, and that the human visual system interprets to provide aperception of depth.

Generating a realistic and comfortable perception of depth ischallenging, however. It will be appreciated that light from objects atdifferent distances from the eyes have wavefronts with different amountsof divergence. FIGS. 3A-3C illustrate relationships between distance andthe divergence of light rays. The distance between the object and theeye 210 is represented by, in order of decreasing distance, R1, R2, andR3. As shown in FIGS. 3A-3C, the light rays become more divergent asdistance to the object decreases. Conversely, as distance increases, thelight rays become more collimated. Stated another way, it may be saidthat the light field produced by a point (the object or a part of theobject) has a spherical wavefront curvature, which is a function of howfar away the point is from the eye of the user. The curvature increaseswith decreasing distance between the object and the eye 210. While onlya single eye 210 is illustrated for clarity of illustration in FIGS.3A-3C and other figures herein, the discussions regarding eye 210 may beapplied to both eyes 210 and 220 of a viewer.

With continued reference to FIGS. 3A-3C, light from an object that theviewer's eyes are fixated on may have different degrees of wavefrontdivergence. Due to the different amounts of wavefront divergence, thelight may be focused differently by the lens of the eye, which in turnmay require the lens to assume different shapes to form a focused imageon the retina of the eye. Where a focused image is not formed on theretina, the resulting retinal blur acts as a cue to accommodation thatcauses a change in the shape of the lens of the eye until a focusedimage is formed on the retina. For example, the cue to accommodation maytrigger the ciliary muscles surrounding the lens of the eye to relax orcontract, thereby modulating the force applied to the suspensoryligaments holding the lens, thus causing the shape of the lens of theeye to change until retinal blur of an object of fixation is eliminatedor minimized, thereby forming a focused image of the object of fixationon the retina (e.g., fovea) of the eye. The process by which the lens ofthe eye changes shape may be referred to as accommodation, and the shapeof the lens of the eye required to form a focused image of the object offixation on the retina (e.g., fovea) of the eye may be referred to as anaccommodative state.

With reference now to FIG. 4A, a representation of theaccommodation-vergence response of the human visual system isillustrated. The movement of the eyes to fixate on an object causes theeyes to receive light from the object, with the light forming an imageon each of the retinas of the eyes. The presence of retinal blur in theimage formed on the retina may provide a cue to accommodation, and therelative locations of the image on the retinas may provide a cue tovergence. The cue to accommodation causes accommodation to occur,resulting in the lenses of the eyes each assuming a particularaccommodative state that forms a focused image of the object on theretina (e.g., fovea) of the eye. On the other hand, the cue to vergencecauses vergence movements (rotation of the eyes) to occur such that theimages formed on each retina of each eye are at corresponding retinalpoints that maintain single binocular vision. In these positions, theeyes may be said to have assumed a particular vergence state. Withcontinued reference to FIG. 4A, accommodation may be understood to bethe process by which the eye achieves a particular accommodative state,and vergence may be understood to be the process by which the eyeachieves a particular vergence state. As indicated in FIG. 4A, theaccommodative and vergence states of the eyes may change if the userfixates on another object. For example, the accommodated state maychange if the user fixates on a new object at a different depth on thez-axis.

Without being limited by theory, it is believed that viewers of anobject may perceive the object as being “three-dimensional” due to acombination of vergence and accommodation. As noted above, vergencemovements (e.g., rotation of the eyes so that the pupils move toward oraway from each other to converge the lines of sight of the eyes tofixate upon an object) of the two eyes relative to each other areclosely associated with accommodation of the lenses of the eyes. Undernormal conditions, changing the shapes of the lenses of the eyes tochange focus from one object to another object at a different distancewill automatically cause a matching change in vergence to the samedistance, under a relationship known as the “accommodation-vergencereflex.” Likewise, a change in vergence will trigger a matching changein lens shape under normal conditions.

With reference now to FIG. 4B, examples of different accommodative andvergence states of the eyes are illustrated. The pair of eyes 222 a arefixated on an object at optical infinity, while the pair eyes 222 b arefixated on an object 221 at less than optical infinity. Notably, thevergence states of each pair of eyes is different, with the pair of eyes222 a directed straight ahead, while the pair of eyes 222 converge onthe object 221. The accommodative states of the eyes forming each pairof eyes 222 a and 222 b are also different, as represented by thedifferent shapes of the lenses 210 a, 220 a.

Undesirably, many users of conventional “3-D” display systems find suchconventional systems to be uncomfortable or may not perceive a sense ofdepth at all due to a mismatch between accommodative and vergence statesin these displays. As noted above, many stereoscopic or “3-D” displaysystems display a scene by providing slightly different images to eacheye. Such systems are uncomfortable for many viewers, since they, amongother things, simply provide different presentations of a scene andcause changes in the vergence states of the eyes, but without acorresponding change in the accommodative states of those eyes. Rather,the images are shown by a display at a fixed distance from the eyes,such that the eyes view all the image information at a singleaccommodative state. Such an arrangement works against the“accommodation-vergence reflex” by causing changes in the vergence statewithout a matching change in the accommodative state. This mismatch isbelieved to cause viewer discomfort. Display systems that provide abetter match between accommodation and vergence may form more realisticand comfortable simulations of three-dimensional imagery.

Without being limited by theory, it is believed that the human eyetypically may interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited numbers of depthplanes. In some embodiments, the different presentations may provideboth cues to vergence and matching cues to accommodation, therebyproviding physiologically correct accommodation-vergence matching.

With continued reference to FIG. 4B, two depth planes 240, correspondingto different distances in space from the eyes 210, 220, are illustrated.For a given depth plane 240, vergence cues may be provided by thedisplaying of images of appropriately different perspectives for eacheye 210, 220. In addition, for a given depth plane 240, light formingthe images provided to each eye 210, 220 may have a wavefront divergencecorresponding to a light field produced by a point at the distance ofthat depth plane 240.

In the illustrated embodiment, the distance, along the z-axis, of thedepth plane 240 containing the point 221 is 1 m. As used herein,distances or depths along the z-axis may be measured with a zero pointlocated at the exit pupils of the user's eyes. Thus, a depth plane 240located at a depth of 1 m corresponds to a distance of 1 m away from theexit pupils of the user's eyes, on the optical axis of those eyes. As anapproximation, the depth or distance along the z-axis may be measuredfrom the display in front of the user's eyes (e.g., from the surface ofa waveguide), plus a value for the distance between the device and theexit pupils of the user's eyes, with the eyes directed towards opticalinfinity. That value may be called the eye relief and corresponds to thedistance between the exit pupil of the user's eye and the display wornby the user in front of the eye. In practice, the value for the eyerelief may be a normalized value used generally for all viewers. Forexample, the eye relief may be assumed to be 20 mm and a depth planethat is at a depth of 1 m may be at a distance of 980 mm in front of thedisplay.

With reference now to FIGS. 4C and 4D, examples of matchedaccommodation-vergence distances and mismatched accommodation-vergencedistances are illustrated, respectively. As illustrated in FIG. 4C, thedisplay system may provide images of a virtual object to each eye 210,220. The images may cause the eyes 210, 220 to assume a vergence statein which the eyes converge on a point 15 on a depth plane 240. Inaddition, the images may be formed by light having a wavefront curvaturecorresponding to real objects at that depth plane 240. As a result, theeyes 210, 220 assume an accommodative state in which the images are infocus on the retinas of those eyes. Thus, the user may perceive thevirtual object as being at the point 15 on the depth plane 240.

It will be appreciated that each of the accommodative and vergencestates of the eyes 210, 220 are associated with a particular distance onthe z-axis. For example, an object at a particular distance from theeyes 210, 220 causes those eyes to assume particular accommodativestates based upon the distances of the object. The distance associatedwith a particular accommodative state may be referred to as theaccommodation distance, A_(d). Similarly, there are particular vergencedistances, V_(d), associated with the eyes in particular vergencestates, or positions relative to one another. Where the accommodationdistance and the vergence distance match, the relationship betweenaccommodation and vergence may be said to be physiologically correct.This is considered to be the most comfortable scenario for a viewer.

In stereoscopic displays, however, the accommodation distance and thevergence distance may not always match. For example, as illustrated inFIG. 4D, images displayed to the eyes 210, 220 may be displayed withwavefront divergence corresponding to depth plane 240, and the eyes 210,220 may assume a particular accommodative state in which the points 15a, 15 b on that depth plane are in focus. However, the images displayedto the eyes 210, 220 may provide cues for vergence that cause the eyes210, 220 to converge on a point 15 that is not located on the depthplane 240. As a result, the accommodation distance corresponds to thedistance from a particular reference point of the user (e.g., the exitpupils of the eyes 210, 220) to the depth plane 240, while the vergencedistance corresponds to the larger distance from that reference point tothe point 15, in some embodiments. Thus, the accommodation distance isdifferent from the vergence distance and there is anaccommodation-vergence mismatch. Such a mismatch is consideredundesirable and may cause discomfort in the user. It will be appreciatedthat the mismatch corresponds to distance (e.g., V_(d)-A_(d)) and may becharacterized using diopters (units of reciprocal length, 1/m). Forexample, a V_(d) of 1.75 diopter and an A_(d) of 1.25 diopter, or aV_(d) of 1.25 diopter and an A_(d) of 1.75 diopter, would provide anaccommodation-vergence mismatch of 0.5 diopter.

In some embodiments, it will be appreciated that a reference point otherthan exit pupils of the eyes 210, 220 may be utilized for determiningdistance for determining accommodation-vergence mismatch, so long as thesame reference point is utilized for the accommodation distance and thevergence distance. For example, the distances could be measured from thecornea to the depth plane, from the retina to the depth plane, from theeyepiece (e.g., a waveguide of the display device) to the depth plane,and so on.

Without being limited by theory, it is believed that users may stillperceive accommodation-vergence mismatches of up to about 0.25 diopter,up to about 0.33 diopter, and up to about 0.5 diopter as beingphysiologically correct, without the mismatch itself causing significantdiscomfort. In some embodiments, display systems disclosed herein (e.g.,the display system 250, FIG. 6 ) present images to the viewer havingaccommodation-vergence mismatch of about 0.5 diopter or less. In someother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.33 diopter or less. In yetother embodiments, the accommodation-vergence mismatch of the imagesprovided by the display system is about 0.25 diopter or less, includingabout 0.1 diopter or less.

FIG. 5 illustrates aspects of an approach for simulatingthree-dimensional imagery by modifying wavefront divergence. The displaysystem includes a waveguide 270 that is configured to receive light 770that is encoded with image information, and to output that light to theuser's eye 210. The waveguide 270 may output the light 650 with adefined amount of wavefront divergence corresponding to the wavefrontdivergence of a light field produced by a point on a desired depth plane240. In some embodiments, the same amount of wavefront divergence isprovided for all objects presented on that depth plane. In addition, itwill be illustrated that the other eye of the user may be provided withimage information from a similar waveguide.

In some embodiments, a single waveguide may be configured to outputlight with a set amount of wavefront divergence corresponding to asingle or limited number of depth planes and/or the waveguide may beconfigured to output light of a limited range of wavelengths.Consequently, in some embodiments, a plurality or stack of waveguidesmay be utilized to provide different amounts of wavefront divergence fordifferent depth planes and/or to output light of different ranges ofwavelengths. As used herein, it will be appreciated at a depth plane mayfollow the contours of a flat or a curved surface. In some embodiments,for simplicity, the depth planes may follow the contours of flatsurfaces.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user. A display system 250 includes a stack ofwaveguides, or stacked waveguide assembly, 260 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 270, 280, 290, 300, 310. It will be appreciated that thedisplay system 250 may be considered a light field display in someembodiments. In addition, the waveguide assembly 260 may also bereferred to as an eyepiece.

In some embodiments, the display system 250 may be configured to providesubstantially continuous cues to vergence and multiple discrete cues toaccommodation. The cues to vergence may be provided by displayingdifferent images to each of the eyes of the user, and the cues toaccommodation may be provided by outputting the light that forms theimages with selectable discrete amounts of wavefront divergence. Statedanother way, the display system 250 may be configured to output lightwith variable levels of wavefront divergence. In some embodiments, eachdiscrete level of wavefront divergence corresponds to a particular depthplane and may be provided by a particular one of the waveguides 270,280, 290, 300, 310.

With continued reference to FIG. 6 , the waveguide assembly 260 may alsoinclude a plurality of features 320, 330, 340, 350 between thewaveguides. In some embodiments, the features 320, 330, 340, 350 may beone or more lenses. The waveguides 270, 280, 290, 300, 310 and/or theplurality of lenses 320, 330, 340, 350 may be configured to send imageinformation to the eye with various levels of wavefront curvature orlight ray divergence. Each waveguide level may be associated with aparticular depth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 360, 370,380, 390, 400 may function as a source of light for the waveguides andmay be utilized to inject image information into the waveguides 270,280, 290, 300, 310, each of which may be configured, as describedherein, to distribute incoming light across each respective waveguide,for output toward the eye 210. Light exits an output surface 410, 420,430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 andis injected into a corresponding input surface 460, 470, 480, 490, 500of the waveguides 270, 280, 290, 300, 310. In some embodiments, each ofthe input surfaces 460, 470, 480, 490, 500 may be an edge of acorresponding waveguide, or may be part of a major surface of thecorresponding waveguide (that is, one of the waveguide surfaces directlyfacing the world 510 or the viewer's eye 210). In some embodiments, asingle beam of light (e.g. a collimated beam) may be injected into eachwaveguide to output an entire field of cloned collimated beams that aredirected toward the eye 210 at particular angles (and amounts ofdivergence) corresponding to the depth plane associated with aparticular waveguide. In some embodiments, a single one of the imageinjection devices 360, 370, 380, 390, 400 may be associated with andinject light into a plurality (e.g., three) of the waveguides 270, 280,290, 300, 310.

In some embodiments, the image injection devices 360, 370, 380, 390, 400are discrete displays that each produce image information for injectioninto a corresponding waveguide 270, 280, 290, 300, 310, respectively. Insome other embodiments, the image injection devices 360, 370, 380, 390,400 are the output ends of a single multiplexed display which may, e.g.,pipe image information via one or more optical conduits (such as fiberoptic cables) to each of the image injection devices 360, 370, 380, 390,400. It will be appreciated that the image information provided by theimage injection devices 360, 370, 380, 390, 400 may include light ofdifferent wavelengths, or colors (e.g., different component colors, asdiscussed herein).

In some embodiments, the light injected into the waveguides 270, 280,290, 300, 310 is provided by a light projector system 520, whichcomprises a light module 530, which may include a light emitter, such asa light emitting diode (LED). The light from the light module 530 may bedirected to and modified by a light modulator 540, e.g., a spatial lightmodulator, via a beam splitter 550. The light modulator 540 may beconfigured to change the perceived intensity of the light injected intothe waveguides 270, 280, 290, 300, 310 to encode the light with imageinformation. Examples of spatial light modulators include liquid crystaldisplays (LCD) including a liquid crystal on silicon (LCOS) displays. Itwill be appreciated that the image injection devices 360, 370, 380, 390,400 are illustrated schematically and, in some embodiments, these imageinjection devices may represent different light paths and locations in acommon projection system configured to output light into associated onesof the waveguides 270, 280, 290, 300, 310. In some embodiments, thewaveguides of the waveguide assembly 260 may function as ideal lenswhile relaying light injected into the waveguides out to the user'seyes. In this conception, the object may be the spatial light modulator540 and the image may be the image on the depth plane.

In some embodiments, the display system 250 may be a scanning fiberdisplay comprising one or more scanning fibers configured to projectlight in various patterns (e.g., raster scan, spiral scan, Lissajouspatterns, etc.) into one or more waveguides 270, 280, 290, 300, 310 andultimately to the eye 210 of the viewer. In some embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a single scanning fiber or a bundle of scanningfibers configured to inject light into one or a plurality of thewaveguides 270, 280, 290, 300, 310. In some other embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a plurality of scanning fibers or a plurality ofbundles of scanning fibers, each of which are configured to inject lightinto an associated one of the waveguides 270, 280, 290, 300, 310. Itwill be appreciated that one or more optical fibers may be configured totransmit light from the light module 530 to the one or more waveguides270, 280, 290, 300, 310. It will be appreciated that one or moreintervening optical structures may be provided between the scanningfiber, or fibers, and the one or more waveguides 270, 280, 290, 300, 310to, e.g., redirect light exiting the scanning fiber into the one or morewaveguides 270, 280, 290, 300, 310.

A controller 560 controls the operation of one or more of the stackedwaveguide assembly 260, including operation of the image injectiondevices 360, 370, 380, 390, 400, the light source 530, and the lightmodulator 540. In some embodiments, the controller 560 is part of thelocal data processing module 140. The controller 560 includesprogramming (e.g., instructions in a non-transitory medium) thatregulates the timing and provision of image information to thewaveguides 270, 280, 290, 300, 310 according to, e.g., any of thevarious schemes disclosed herein. In some embodiments, the controllermay be a single integral device, or a distributed system connected bywired or wireless communication channels. The controller 560 may be partof the processing modules 140 or 150 (FIG. 9D) in some embodiments.

With continued reference to FIG. 6 , the waveguides 270, 280, 290, 300,310 may be configured to propagate light within each respectivewaveguide by total internal reflection (TIR). The waveguides 270, 280,290, 300, 310 may each be planar or have another shape (e.g., curved),with major top and bottom surfaces and edges extending between thosemajor top and bottom surfaces. In the illustrated configuration, thewaveguides 270, 280, 290, 300, 310 may each include out-coupling opticalelements 570, 580, 590, 600, 610 that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 210. Extracted light may also be referred to as out-coupledlight and the out-coupling optical elements light may also be referredto light extracting optical elements. An extracted beam of light may beoutputted by the waveguide at locations at which the light propagatingin the waveguide strikes a light extracting optical element. Theout-coupling optical elements 570, 580, 590, 600, 610 may, for example,be gratings, including diffractive optical features, as discussedfurther herein. While illustrated disposed at the bottom major surfacesof the waveguides 270, 280, 290, 300, 310, for ease of description anddrawing clarity, in some embodiments, the out-coupling optical elements570, 580, 590, 600, 610 may be disposed at the top and/or bottom majorsurfaces, and/or may be disposed directly in the volume of thewaveguides 270, 280, 290, 300, 310, as discussed further herein. In someembodiments, the out-coupling optical elements 570, 580, 590, 600, 610may be formed in a layer of material that is attached to a transparentsubstrate to form the waveguides 270, 280, 290, 300, 310. In some otherembodiments, the waveguides 270, 280, 290, 300, 310 may be a monolithicpiece of material and the out-coupling optical elements 570, 580, 590,600, 610 may be formed on a surface and/or in the interior of that pieceof material.

With continued reference to FIG. 6 , as discussed herein, each waveguide270, 280, 290, 300, 310 is configured to output light to form an imagecorresponding to a particular depth plane. For example, the waveguide270 nearest the eye may be configured to deliver collimated light (whichwas injected into such waveguide 270), to the eye 210. The collimatedlight may be representative of the optical infinity focal plane. Thenext waveguide up 280 may be configured to send out collimated lightwhich passes through the first lens 350 (e.g., a negative lens) beforeit may reach the eye 210; such first lens 350 may be configured tocreate a slight convex wavefront curvature so that the eye/braininterprets light coming from that next waveguide up 280 as coming from afirst focal plane closer inward toward the eye 210 from opticalinfinity. Similarly, the third up waveguide 290 passes its output lightthrough both the first 350 and second 340 lenses before reaching the eye210; the combined optical power of the first 350 and second 340 lensesmay be configured to create another incremental amount of wavefrontcurvature so that the eye/brain interprets light coming from the thirdwaveguide 290 as coming from a second focal plane that is even closerinward toward the person from optical infinity than was light from thenext waveguide up 280.

The other waveguide layers 300, 310 and lenses 330, 320 are similarlyconfigured, with the highest waveguide 310 in the stack sending itsoutput through all of the lenses between it and the eye for an aggregatefocal power representative of the closest focal plane to the person. Tocompensate for the stack of lenses 320, 330, 340, 350 whenviewing/interpreting light coming from the world 510 on the other sideof the stacked waveguide assembly 260, a compensating lens layer 620 maybe disposed at the top of the stack to compensate for the aggregatepower of the lens stack 320, 330, 340, 350 below. Such a configurationprovides as many perceived focal planes as there are availablewaveguide/lens pairings. Both the out-coupling optical elements of thewaveguides and the focusing aspects of the lenses may be static (i.e.,not dynamic or electro-active). In some alternative embodiments, eitheror both may be dynamic using electro-active features.

In some embodiments, two or more of the waveguides 270, 280, 290, 300,310 may have the same associated depth plane. For example, multiplewaveguides 270, 280, 290, 300, 310 may be configured to output imagesset to the same depth plane, or multiple subsets of the waveguides 270,280, 290, 300, 310 may be configured to output images set to the sameplurality of depth planes, with one set for each depth plane. This mayprovide advantages for forming a tiled image to provide an expandedfield of view at those depth planes.

With continued reference to FIG. 6 , the out-coupling optical elements570, 580, 590, 600, 610 may be configured to both redirect light out oftheir respective waveguides and to output this light with theappropriate amount of divergence or collimation for a particular depthplane associated with the waveguide. As a result, waveguides havingdifferent associated depth planes may have different configurations ofout-coupling optical elements 570, 580, 590, 600, 610, which outputlight with a different amount of divergence depending on the associateddepth plane. In some embodiments, the light extracting optical elements570, 580, 590, 600, 610 may be volumetric or surface features, which maybe configured to output light at specific angles. For example, the lightextracting optical elements 570, 580, 590, 600, 610 may be volumeholograms, surface holograms, and/or diffraction gratings. In someembodiments, the features 320, 330, 340, 350 may not be lenses; rather,they may simply be spacers (e.g., cladding layers and/or structures forforming air gaps).

In some embodiments, the out-coupling optical elements 570, 580, 590,600, 610 are diffractive features that form a diffraction pattern, or“diffractive optical element” (also referred to herein as a “DOE”).Preferably, the DOE's have a sufficiently low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye 210 with each intersection of the DOE, while the rest continuesto move through a waveguide via TIR. The light carrying the imageinformation is thus divided into a number of related exit beams thatexit the waveguide at a multiplicity of locations and the result is afairly uniform pattern of exit emission toward the eye 210 for thisparticular collimated beam bouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets may be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet may be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, a camera assembly 630 (e.g., a digital camera,including visible light and infrared light cameras) may be provided tocapture images of the eye 210 and/or tissue around the eye 210 to, e.g.,detect user inputs and/or to monitor the physiological state of theuser. As used herein, a camera may be any image capture device. In someembodiments, the camera assembly 630 may include an image capture deviceand a light source to project light (e.g., infrared light) to the eye,which may then be reflected by the eye and detected by the image capturedevice. In some embodiments, the camera assembly 630 may be attached tothe frame 80 (FIG. 9D) and may be in electrical communication with theprocessing modules 140 and/or 150, which may process image informationfrom the camera assembly 630. In some embodiments, one camera assembly630 may be utilized for each eye, to separately monitor each eye.

With reference now to FIG. 7 , an example of exit beams outputted by awaveguide is shown. One waveguide is illustrated, but it will beappreciated that other waveguides in the waveguide assembly 260 (FIG. 6) may function similarly, where the waveguide assembly 260 includesmultiple waveguides. Light 640 is injected into the waveguide 270 at theinput surface 460 of the waveguide 270 and propagates within thewaveguide 270 by TIR. At points where the light 640 impinges on the DOE570, a portion of the light exits the waveguide as exit beams 650. Theexit beams 650 are illustrated as substantially parallel but, asdiscussed herein, they may also be redirected to propagate to the eye210 at an angle (e.g., forming divergent exit beams), depending on thedepth plane associated with the waveguide 270. It will be appreciatedthat substantially parallel exit beams may be indicative of a waveguidewith out-coupling optical elements that out-couple light to form imagesthat appear to be set on a depth plane at a large distance (e.g.,optical infinity) from the eye 210. Other waveguides or other sets ofout-coupling optical elements may output an exit beam pattern that ismore divergent, which would require the eye 210 to accommodate to acloser distance to bring it into focus on the retina and would beinterpreted by the brain as light from a distance closer to the eye 210than optical infinity.

In some embodiments, a full color image may be formed at each depthplane by overlaying images in each of the component colors, e.g., threeor more component colors. FIG. 8 illustrates an example of a stackedwaveguide assembly in which each depth plane includes images formedusing multiple different component colors. The illustrated embodimentshows depth planes 240 a-240 f, although more or fewer depths are alsocontemplated. Each depth plane may have three or more component colorimages associated with it, including: a first image of a first color, G;a second image of a second color, R; and a third image of a third color,B. Different depth planes are indicated in the figure by differentnumbers for diopters (dpt) following the letters G, R, and B. Just asexamples, the numbers following each of these letters indicate diopters(1/m), or inverse distance of the depth plane from a viewer, and eachbox in the figures represents an individual component color image. Insome embodiments, to account for differences in the eye's focusing oflight of different wavelengths, the exact placement of the depth planesfor different component colors may vary. For example, differentcomponent color images for a given depth plane may be placed on depthplanes corresponding to different distances from the user. Such anarrangement may increase visual acuity and user comfort and/or maydecrease chromatic aberrations.

In some embodiments, light of each component color may be outputted by asingle dedicated waveguide and, consequently, each depth plane may havemultiple waveguides associated with it. In such embodiments, each box inthe figures including the letters G, R, or B may be understood torepresent an individual waveguide, and three waveguides may be providedper depth plane where three component color images are provided perdepth plane. While the waveguides associated with each depth plane areshown adjacent to one another in this drawing for ease of description,it will be appreciated that, in a physical device, the waveguides mayall be arranged in a stack with one waveguide per level. In some otherembodiments, multiple component colors may be outputted by the samewaveguide, such that, e.g., only a single waveguide may be provided perdepth plane.

With continued reference to FIG. 8 , in some embodiments, G is the colorgreen, R is the color red, and B is the color blue. In some otherembodiments, other colors associated with other wavelengths of light,including magenta and cyan, may be used in addition to or may replaceone or more of red, green, or blue.

It will be appreciated that references to a given color of lightthroughout this disclosure will be understood to encompass light of oneor more wavelengths within a range of wavelengths of light that areperceived by a viewer as being of that given color. For example, redlight may include light of one or more wavelengths in the range of about620-780 nm, green light may include light of one or more wavelengths inthe range of about 492-577 nm, and blue light may include light of oneor more wavelengths in the range of about 435-493 nm.

In some embodiments, the light source 530 (FIG. 6 ) may be configured toemit light of one or more wavelengths outside the visual perceptionrange of the viewer, for example, infrared and/or ultravioletwavelengths. In addition, the in-coupling, out-coupling, and other lightredirecting structures of the waveguides of the display 250 may beconfigured to direct and emit this light out of the display towards theuser's eye 210, e.g., for imaging and/or user stimulation applications.

With reference now to FIG. 9A, in some embodiments, light impinging on awaveguide may need to be redirected to in-couple that light into thewaveguide. An in-coupling optical element may be used to redirect andin-couple the light into its corresponding waveguide. FIG. 9Aillustrates a cross-sectional side view of an example of a plurality orset 660 of stacked waveguides that each includes an in-coupling opticalelement. The waveguides may each be configured to output light of one ormore different wavelengths, or one or more different ranges ofwavelengths. It will be appreciated that the stack 660 may correspond tothe stack 260 (FIG. 6 ) and the illustrated waveguides of the stack 660may correspond to part of the plurality of waveguides 270, 280, 290,300, 310, except that light from one or more of the image injectiondevices 360, 370, 380, 390, 400 is injected into the waveguides from aposition that requires light to be redirected for in-coupling.

The illustrated set 660 of stacked waveguides includes waveguides 670,680, and 690. Each waveguide includes an associated in-coupling opticalelement (which may also be referred to as a light input area on thewaveguide), with, e.g., in-coupling optical element 700 disposed on amajor surface (e.g., an upper major surface) of waveguide 670,in-coupling optical element 710 disposed on a major surface (e.g., anupper major surface) of waveguide 680, and in-coupling optical element720 disposed on a major surface (e.g., an upper major surface) ofwaveguide 690. In some embodiments, one or more of the in-couplingoptical elements 700, 710, 720 may be disposed on the bottom majorsurface of the respective waveguide 670, 680, 690 (particularly wherethe one or more in-coupling optical elements are reflective, deflectingoptical elements). As illustrated, the in-coupling optical elements 700,710, 720 may be disposed on the upper major surface of their respectivewaveguide 670, 680, 690 (or the top of the next lower waveguide),particularly where those in-coupling optical elements are transmissive,deflecting optical elements. In some embodiments, the in-couplingoptical elements 700, 710, 720 may be disposed in the body of therespective waveguide 670, 680, 690. In some embodiments, as discussedherein, the in-coupling optical elements 700, 710, 720 are wavelengthselective, such that they selectively redirect one or more wavelengthsof light, while transmitting other wavelengths of light. Whileillustrated on one side or corner of their respective waveguide 670,680, 690, it will be appreciated that the in-coupling optical elements700, 710, 720 may be disposed in other areas of their respectivewaveguide 670, 680, 690 in some embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 6 , and may be separated (e.g., laterallyspaced apart) from other in-coupling optical elements 700, 710, 720 suchthat it substantially does not receive light from the other ones of thein-coupling optical elements 700, 710, 720.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750, may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750, may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690, respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, and/or solid layers of material. For example, asillustrated, layer 760 a may separate waveguides 670 and 680; and layer760 b may separate waveguides 680 and 690. In some embodiments, thelayers 760 a and 760 b are formed of low refractive index materials(that is, materials having a lower refractive index than the materialforming the immediately adjacent one of waveguides 670, 680, 690).Preferably, the refractive index of the material forming the layers 760a, 760 b is 0.05 or more, or 0.10 or less than the refractive index ofthe material forming the waveguides 670, 680, 690. Advantageously, thelower refractive index layers 760 a, 760 b may function as claddinglayers that facilitate total internal reflection (TIR) of light throughthe waveguides 670, 680, 690 (e.g., TIR between the top and bottom majorsurfaces of each waveguide). In some embodiments, the layers 760 a, 760b are formed of air. While not illustrated, it will be appreciated thatthe top and bottom of the illustrated set 660 of waveguides may includeimmediately neighboring cladding layers.

Preferably, for ease of manufacturing and other considerations, thematerial forming the waveguides 670, 680, 690 are similar or the same,and the material forming the layers 760 a, 760 b are similar or thesame. In some embodiments, the material forming the waveguides 670, 680,690 may be different between one or more waveguides, and/or the materialforming the layers 760 a, 760 b may be different, while still holding tothe various refractive index relationships noted above.

With continued reference to FIG. 9A, light rays 770, 780, 790 areincident on the set 660 of waveguides. It will be appreciated that thelight rays 770, 780, 790 may be injected into the waveguides 670, 680,690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG.6 ).

In some embodiments, the light rays 770, 780, 790 have differentproperties, e.g., different wavelengths or different ranges ofwavelengths, which may correspond to different colors. The in-couplingoptical elements 700, 710, 720 each deflect the incident light such thatthe light propagates through a respective one of the waveguides 670,680, 690 by TIR. In some embodiments, the incoupling optical elements700, 710, 720 each selectively deflect one or more particularwavelengths of light, while transmitting other wavelengths to anunderlying waveguide and associated incoupling optical element.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths,while transmitting rays 780 and 790, which have different second andthird wavelengths or ranges of wavelengths, respectively. Thetransmitted ray 780 impinges on and is deflected by the in-couplingoptical element 710, which is configured to deflect light of a secondwavelength or range of wavelengths. The ray 790 is deflected by thein-coupling optical element 720, which is configured to selectivelydeflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 9A, the deflected light rays 770, 780,790 are deflected so that they propagate through a correspondingwaveguide 670, 680, 690; that is, the in-coupling optical elements 700,710, 720 of each waveguide deflects light into that correspondingwaveguide 670, 680, 690 to in-couple light into that correspondingwaveguide. The light rays 770, 780, 790 are deflected at angles thatcause the light to propagate through the respective waveguide 670, 680,690 by TIR. The light rays 770, 780, 790 propagate through therespective waveguide 670, 680, 690 by TIR until impinging on thewaveguide's corresponding light distributing elements 730, 740, 750.

With reference now to FIG. 9B, a perspective view of an example of theplurality of stacked waveguides of FIG. 9A is illustrated. As notedabove, the in-coupled light rays 770, 780, 790, are deflected by thein-coupling optical elements 700, 710, 720, respectively, and thenpropagate by TIR within the waveguides 670, 680, 690, respectively. Thelight rays 770, 780, 790 then impinge on the light distributing elements730, 740, 750, respectively. The light distributing elements 730, 740,750 deflect the light rays 770, 780, 790 so that they propagate towardsthe out-coupling optical elements 800, 810, 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE's). In some embodiments, the OPE'sdeflect or distribute light to the out-coupling optical elements 800,810, 820 and, in some embodiments, may also increase the beam or spotsize of this light as it propagates to the out-coupling opticalelements. In some embodiments, the light distributing elements 730, 740,750 may be omitted and the in-coupling optical elements 700, 710, 720may be configured to deflect light directly to the out-coupling opticalelements 800, 810, 820. For example, with reference to FIG. 9A, thelight distributing elements 730, 740, 750 may be replaced without-coupling optical elements 800, 810, 820, respectively. In someembodiments, the out-coupling optical elements 800, 810, 820 are exitpupils (EP's) or exit pupil expanders (EPE's) that direct light in aviewer's eye 210 (FIG. 7 ). It will be appreciated that the OPE's may beconfigured to increase the dimensions of the eye box in at least oneaxis and the EPE's may be to increase the eye box in an axis crossing,e.g., orthogonal to, the axis of the OPEs. For example, each OPE may beconfigured to redirect a portion of the light striking the OPE to an EPEof the same waveguide, while allowing the remaining portion of the lightto continue to propagate down the waveguide. Upon impinging on the OPEagain, another portion of the remaining light is redirected to the EPE,and the remaining portion of that portion continues to propagate furtherdown the waveguide, and so on. Similarly, upon striking the EPE, aportion of the impinging light is directed out of the waveguide towardsthe user, and a remaining portion of that light continues to propagatethrough the waveguide until it strikes the EP again, at which timeanother portion of the impinging light is directed out of the waveguide,and so on. Consequently, a single beam of incoupled light may be“replicated” each time a portion of that light is redirected by an OPEor EPE, thereby forming a field of cloned beams of light, as shown inFIG. 6 . In some embodiments, the OPE and/or EPE may be configured tomodify a size of the beams of light.

Accordingly, with reference to FIGS. 9A and 9B, in some embodiments, theset 660 of waveguides includes waveguides 670, 680, 690; in-couplingoptical elements 700, 710, 720; light distributing elements (e.g.,OPE's) 730, 740, 750; and out-coupling optical elements (e.g., EP's)800, 810, 820 for each component color. The waveguides 670, 680, 690 maybe stacked with an air gap/cladding layer between each one. Thein-coupling optical elements 700, 710, 720 redirect or deflect incidentlight (with different in-coupling optical elements receiving light ofdifferent wavelengths) into its waveguide. The light then propagates atan angle which will result in TIR within the respective waveguide 670,680, 690. In the example shown, light ray 770 (e.g., blue light) isdeflected by the first in-coupling optical element 700, and thencontinues to bounce down the waveguide, interacting with the lightdistributing element (e.g., OPE's) 730 and then the out-coupling opticalelement (e.g., EPs) 800, in a manner described earlier. The light rays780 and 790 (e.g., green and red light, respectively) will pass throughthe waveguide 670, with light ray 780 impinging on and being deflectedby in-coupling optical element 710. The light ray 780 then bounces downthe waveguide 680 via TIR, proceeding on to its light distributingelement (e.g., OPEs) 740 and then the out-coupling optical element(e.g., EP's) 810. Finally, light ray 790 (e.g., red light) passesthrough the waveguide 690 to impinge on the light in-coupling opticalelements 720 of the waveguide 690. The light in-coupling opticalelements 720 deflect the light ray 790 such that the light raypropagates to light distributing element (e.g., OPEs) 750 by TIR, andthen to the out-coupling optical element (e.g., EPs) 820 by TIR. Theout-coupling optical element 820 then finally out-couples the light ray790 to the viewer, who also receives the out-coupled light from theother waveguides 670, 680.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B. As illustrated, the waveguides670, 680, 690, along with each waveguide's associated light distributingelement 730, 740, 750 and associated out-coupling optical element 800,810, 820, may be vertically aligned. However, as discussed herein, thein-coupling optical elements 700, 710, 720 are not vertically aligned;rather, the in-coupling optical elements are preferably non-overlapping(e.g., laterally spaced apart as seen in the top-down view). Asdiscussed further herein, this nonoverlapping spatial arrangementfacilitates the injection of light from different resources intodifferent waveguides on a one-to-one basis, thereby allowing a specificlight source to be uniquely coupled to a specific waveguide. In someembodiments, arrangements including nonoverlapping spatially-separatedin-coupling optical elements may be referred to as a shifted pupilsystem, and the in-coupling optical elements within these arrangementsmay correspond to sub pupils.

FIG. 9D illustrates an example of wearable display system 60 into whichthe various waveguides and related systems disclosed herein may beintegrated. In some embodiments, the display system 60 is the system 250of FIG. 6 , with FIG. 6 schematically showing some parts of that system60 in greater detail. For example, the waveguide assembly 260 of FIG. 6may be part of the display 70.

With continued reference to FIG. 9D, the display system 60 includes adisplay 70, and various mechanical and electronic modules and systems tosupport the functioning of that display 70. The display 70 may becoupled to a frame 80, which is wearable by a display system user orviewer 90 and which is configured to position the display 70 in front ofthe eyes of the user 90. The display 70 may be considered eyewear insome embodiments. In some embodiments, a speaker 100 is coupled to theframe 80 and configured to be positioned adjacent the ear canal of theuser 90 (in some embodiments, another speaker, not shown, may optionallybe positioned adjacent the other ear canal of the user to providestereo/shapeable sound control). The display system 60 may also includeone or more microphones 110 or other devices to detect sound. In someembodiments, the microphone is configured to allow the user to provideinputs or commands to the system 60 (e.g., the selection of voice menucommands, natural language questions, etc.), and/or may allow audiocommunication with other persons (e.g., with other users of similardisplay systems. The microphone may further be configured as aperipheral sensor to collect audio data (e.g., sounds from the userand/or environment). In some embodiments, the display system 60 mayfurther include one or more outwardly-directed environmental sensors 112configured to detect light, objects, stimuli, people, animals,locations, or other aspects of the world around the user. For example,environmental sensors 112 may include one or more cameras, which may belocated, for example, facing outward so as to capture images similar toat least a portion of an ordinary field of view of the user 90. In someembodiments, the display system may also include a peripheral sensor 120a, which may be separate from the frame 80 and attached to the body ofthe user 90 (e.g., on the head, torso, an extremity, etc. of the user90). The peripheral sensor 120 a may be configured to acquire datacharacterizing a physiological state of the user 90 in some embodiments.For example, the sensor 120 a may be an electrode.

With continued reference to FIG. 9D, the display 70 is operativelycoupled by communications link 130, such as by a wired lead or wirelessconnectivity, to a local data processing module 140 which may be mountedin a variety of configurations, such as fixedly attached to the frame80, fixedly attached to a helmet or hat worn by the user, embedded inheadphones, or otherwise removably attached to the user 90 (e.g., in abackpack-style configuration, in a belt-coupling style configuration).Similarly, the sensor 120 a may be operatively coupled by communicationslink 120 b, e.g., a wired lead or wireless connectivity, to the localprocessor and data module 140. The local processing and data module 140may comprise a hardware processor, as well as digital memory, such asnon-volatile memory (e.g., flash memory or hard disk drives), both ofwhich may be utilized to assist in the processing, caching, and storageof data. Optionally, the local processor and data module 140 may includeone or more central processing units (CPUs), graphics processing units(GPUs), dedicated processing hardware, and so on. The data may includedata a) captured from sensors (which may be, e.g., operatively coupledto the frame 80 or otherwise attached to the user 90), such as imagecapture devices (such as cameras), microphones, inertial measurementunits, accelerometers, compasses, GPS units, radio devices, gyros,and/or other sensors disclosed herein; and/or b) acquired and/orprocessed using remote processing module 150 and/or remote datarepository 160 (including data relating to virtual content), possiblyfor passage to the display 70 after such processing or retrieval. Thelocal processing and data module 140 may be operatively coupled bycommunication links 170, 180, such as via a wired or wirelesscommunication links, to the remote processing module 150 and remote datarepository 160 such that these remote modules 150, 160 are operativelycoupled to each other and available as resources to the local processingand data module 140. In some embodiments, the local processing and datamodule 140 may include one or more of the image capture devices,microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, and/or gyros. In some other embodiments, one ormore of these sensors may be attached to the frame 80, or may bestandalone structures that communicate with the local processing anddata module 140 by wired or wireless communication pathways.

With continued reference to FIG. 9D, in some embodiments, the remoteprocessing module 150 may comprise one or more processors configured toanalyze and process data and/or image information, for instanceincluding one or more central processing units (CPUs), graphicsprocessing units (GPUs), dedicated processing hardware, and so on. Insome embodiments, the remote data repository 160 may comprise a digitaldata storage facility, which may be available through the internet orother networking configuration in a “cloud” resource configuration. Insome embodiments, the remote data repository 160 may include one or moreremote servers, which provide information, e.g., information forgenerating augmented reality content, to the local processing and datamodule 140 and/or the remote processing module 150. In some embodiments,all data is stored and all computations are performed in the localprocessing and data module, allowing fully autonomous use from a remotemodule. Optionally, an outside system (e.g., a system of one or moreprocessors, one or more computers) that includes CPUs, GPUs, and so on,may perform at least a portion of processing (e.g., generating imageinformation, processing data) and provide information to, and receiveinformation from, modules 140, 150, 160, for instance via wireless orwired connections.

I. Adjusting Quality Based on Depth Information

As described herein, display systems (e.g., augmented reality displaysystems such as the display system 60, FIG. 9D) according to variousembodiments may determine a three-dimensional fixation point of theuser, e.g., by monitoring a user's eyes. The fixation point may indicatethe location of the point in space along (1) an x-axis (e.g., a lateralaxis), (2) a y-axis (e.g., a vertical axis), and (3) a z-axis (e.g., adepth of the point, for example a depth from the user). In someembodiments, the display system may utilize cameras, sensors, and so on,to monitor the user's eyes (e.g., a pupil, cornea, and so on, of eacheye), to determine a gaze of each eye. The gaze of each eye may beunderstood to be a vector extending from generally a center of theretina of that eye through the lens of the eye. For example, the vectormay extend generally from the center of the macula (e.g., the fovea)through the lens of the eye. The display system may be configured todetermine where the vectors associated with the eyes intersect, and thisintersection point may be understood to be the fixation point of theeyes. Stated another way, the fixation point may be location inthree-dimensional space on which the user's eyes are verging. In someembodiments, the display system may filter small movements of the user'seyes for example during rapid movements (e.g., saccades, microsaccades),and may update the fixation point upon determining that the eyes arefixating on a location in three-dimensional space. For example, thedisplay system may be configured to ignore movements of the eye thatfixate on a point for less than a threshold duration.

The resolution of content presented by the display system, such asvirtual objects or content, may be adjusted based on proximity to thefixation point as discussed herein. It will be appreciated that thedisplay system may have stored within it, or may have access to,information regarding the locations, in three-dimensional space, ofvirtual objects. Based on the known locations of the virtual objects,the proximity of a given virtual object to the fixation point may bedetermined. For example, the proximity of the virtual object to thefixation point may be determined by determining one or more of the (1)three-dimensional distance of a virtual object from the fixation pointof the user; (2) the resolution adjustment zone in which the virtualobject is located, relative to the resolution adjustment zone in whichthe fixation point is located, in cases where the display system'sdisplay frustum is divided into resolution adjustment zones; and (3) theangular separation between the virtual object and a gaze of the user.Virtual content that is closer in proximity to the fixation point may bepresented at a greater resolution than content farther from the fixationpoint. In some embodiments, the resolution of virtual content changesdepending upon the proximity of the depth plane on which that virtualcontent is disposed to the fixation point or the depth plane on whichthe fixation point is disposed. In some embodiments, adjustments to theresolution may be made by a rendering engine, such as rendering enginesincluded in one or more graphics processing units, for instance in oneor more of modules 140, 150 (FIG. 9D).

FIG. 10A illustrates an example of a representation of a top-down viewof a user viewing content (e.g., content included in a display frustum1004) presented by a display system (e.g., the display system 60, FIG.9D). The representation includes the user's eyes 210, 220, and adetermination of a fixation point 1006 of the eyes 210, 220. Asillustrated, the gaze of each eye is represented as a vector (e.g.,vectors 1003A, 1003B) and the display system has detected the fixationpoint 1006 by, e.g., determining where those vectors converge in frontof the eyes 210, 22. In the illustrated example, the fixation point 1006coincides with the location of a first virtual object 1008A presented bythe display system. Examples of systems and methods for eye-tracking maybe found in U.S. application Ser. No. 14/690,401 filed on Apr. 18, 2015,which is incorporated by reference for all of purposes; and in theattached Appendix. For example, eye-tracking systems and methods aredescribed in, at least, FIGS. 25-27 of the Appendix, and can beutilized, at least in part, for eye-tracking and/or to determinefixation points as described herein.

With continued reference to FIG. 10A, a second virtual object 1008B isalso presented by the display system in the display frustum 1004. Theview of these virtual objects 1008A, 1008B, as seen by the viewer, isshown in a rendered frame 1010. The rendered frame 1010 may include thefirst virtual object 1008A rendered at a first resolution, while thesecond virtual object 1008B, located away from the fixation point 1006,is rendered at a second, lesser resolution. Specifically, the secondvirtual object 1008B may be determined to be located at a greater depththan, and towards the side of, the first virtual object 1008A. Forexample, the display system may determine the depth of the secondvirtual object 1008B, as discussed herein, or optionally a contentprovider associated with the virtual content may indicate depths ofvirtual objects which the display system may utilize for rendering thatvirtual object. Therefore, the fixation point 1006, as described above,describes a three-dimensional location in space at which the user islooking, and the second virtual object 1008B may be determined to belocated further in depth from the user along with being laterallydisplaced from the fixation point 1006.

Without being limited by theory, it is believed that, with the user'seyes 210, 220 looking at the first virtual object 1008A, an image of thefirst virtual object 1008A may fall on the user's fovea, while an imageof the second virtual object 1008B does not fall on the fovea. As aresult, the second virtual object 1008B may be reduced in resolutionwithout significant impact to the perceived image quality of the displaysystem, due to a lower sensitivity of the human visual system to thatsecond virtual object 1008B. In addition, the lower resolutionadvantageously reduces the computational load required to provide theimages. As discussed herein, the resolution at which the second virtualobject 1008B is rendered may be based on a proximity to the fixationpoint 1006, and the reduction in resolution (e.g., with respect to theresolution of the first virtual object 1008A) may increase withdecreasing proximity (or increasing distance) between the fixation point1006 and the virtual object 1008A. In some embodiments, the rate ofdecrease of the resolution may be in conformance with a rate ofreduction of the density of cones in the human eye, or with a visualacuity drop-off away from the fovea.

It will be appreciated that the resolutions of the various virtualobjects presented by the display system may vary dynamically as thefixation point changes location. For example, FIG. 10B illustratesanother example of a representation of a top-down view of a user viewingcontent presented by the display system. As illustrated in FIG. 10B, theuser is now focusing on the second virtual object 1008B, as compared toFIG. 10A, in which the user was focusing on the first virtual object1008A. By monitoring the gaze 1003A, 1003B of the user, the displaysystem determines that the eyes 210, 220 are verging on the secondvirtual object 1008B, and sets that location as the new fixation point1006.

Upon detecting this change in the location of the fixation point 1006,the display system now renders second virtual object 1008B at a greaterresolution than the first virtual object 1008A, as shown in the renderedframe 1010. Preferably, the display system monitors the user's gaze1003A, 1003B at a sufficiently high frequency, and changes theresolution of virtual objects sufficiently quickly, that the transitionin resolution of the first virtual object 1008A and second virtualobject 1008B is substantially imperceptible to the user.

FIG. 10C illustrates another example of a representation of a top-downview of a user viewing content via a display system (e.g., the displaysystem 60, FIG. 9D). In the example, the user's field of view 1004 isillustrated along with a fixation point 1006. Three virtual objects areillustrated, with a first virtual object 1012A being closer in proximityto the fixation point 1006 than a second virtual object 1012B or a thirdvirtual object 1012C. Similarly, the second virtual object 1012B isillustrated as being closer in proximity to the fixation point 1006 thanthe third virtual object 1012C. Therefore, when the virtual objects1012A-1012C are presented to the user, the display system may allocateresources such that rendering the first virtual object 1012A is accordeda greater resource allocation (e.g., the object 1012A is rendered at agreater resolution) than the second virtual object 1012B, and the secondvirtual object 1012B receives a greater resource allocation than thethird virtual object 1012C. The third virtual object 1012C mayoptionally not be rendered at all, as it is outside of the field of view1004.

Resolution adjustment zones are illustrated in the example of FIG. 100 ,with the zones being ellipses (e.g., circles) described along depth andlateral axes. As illustrated, the fixation point 1006 is inside a centerzone 1014A, with the first virtual object 1012A extending between zones1014B, 1014C and within the user's cone 1004 a of foveal vision. Thefirst virtual object 1012A may therefore be presented to the user at aresolution associated with zone 1014B or 1014C, or optionally a portionof the object 1012A within zone 1014B may be presented according to theresolution of zone 1014B and remaining portion within zone 1014C may bepresented according to the resolution of zone 1014C. For example, in anembodiment in which the zones are assigned resolutions reduced from amaximum (e.g., highest) resolution, the first virtual object 1012A maybe presented at the assigned resolutions. Optionally, the first virtualobject 1012A may be presented at either of the resolutions (e.g., thedisplay system may be programmed to display at the highest revolutionassociated with any zones across which the first virtual object 1012Aextends), or a measure of central tendency of the resolutions (e.g., themeasure can be weighted according to an extent to which the object 1012Ais located within the zones 1014B, 1014C). With continued reference toFIG. 100 , it will be appreciated that the resolution adjustment zonesat different distances from the fixation point 1006 may have differentshapes. For example, the zone 1014C may have a different shape from thezones 1014A-1014C, and conform to the contours of the field of view1004. In some other embodiments, one or more of the zones 1014A-1014Cmay have different shapes from one or more others of the zones1014A-1014C.

FIG. 10D is a block diagram of an example display system. The exampledisplay system (e.g., the display system 60, FIG. 9D) may be anaugmented reality display system and/or a mixed reality display system,which can adjust usage of rendering hardware resources according to auser's fixation point as described herein. For example, as describedabove with respect to FIG. 100 , rendering hardware resources 1021 canbe adjusted according to the user's fixation point. A resource arbiter1020 may be implemented to regulate usage of such resources 1021, forexample the arbiter 1020 can allocate the resources 1021 to particularapplication processes 1022 associated with presenting virtual objects tothe user. The resource arbiter 1020 and/or rendering hardware resources1021 may optionally be included in the local processing & data module140 (e.g., as illustrated in FIG. 9D), and/or the remote processingmodule 150, of the display system 60. For example, the renderinghardware resources 1021 may comprise graphics processing units (GPUs),which may be included in module 140 and/or module 150 as described abovewith respect to FIG. 9D.

As an example of adjusting resources 1021, and with respect to FIG. 100, a first virtual object 1012A associated with a first applicationprocess can be allocated a greater share of resources 1021 than a secondvirtual object 1012B associated with a second application process.Virtual objects associated with the application processes 1022 can berendered based on the allocated resources 1021, and included in framebuffers 1024 to be composited (e.g., by compositor 1026) into a finalframe buffer 1028. The final frame buffer 1028 can then be presented bydisplay hardware 1030, for example the display 70 illustrated in FIG.9D, with the rendered virtual objects adjusted in resolution.

As disclosed herein, the resolution of a virtual object may bedetermined based upon the proximity of the virtual object to thefixation point. In some embodiments, the resolution may be modified as afunction of the distance between the virtual object and the fixationpoint. In some embodiments, the modifications may occur in discretesteps; that is, a similar modification may be applied to all virtualobjects disposed in a particular volume or zone. FIG. 11A1 illustratesan example of a representation of a top-down view of adjustments inresolution in different resolution adjustment zones based onthree-dimensional fixation point tracking. The display system may dividethe display frustum into multiple volumes or resolution adjustmentzones, and modify resolution in discrete steps corresponding to thesezones. Thus, in some embodiments, to determine an adjustment in theresolution of virtual content, the display system may utilizeinformation describing volumes of space (referred hereinafter asresolution adjustment zones), and assignments of resolution adjustmentsto each volume of space. As illustrated, a field of view provided by thedisplay system (e.g., the display frustum of the display) is separatedinto a plurality of different zones each encompassing a range of depthsfrom a user (e.g., depth ranges 1102A-1102E). In some embodiments, eachdepth range 1102A-1102E has a single associated depth plane that may bepresented by the display system. With continued reference to FIG. 11A1,five zones encompass each identified range of depths from the user andare contiguous along a lateral direction. In the illustrated exampletop-down view, the field of view is divided into a grid 1100 of 25zones. Each zone represents a volume of real-world space in whichvirtual content may be placed for a user.

It will be appreciated that the zones may also extend in a verticaldirection (e.g., along the y-axis, not shown), such that the illustratedgrid 1100 may be understood to represent one cross-section along thisvertical direction. In some embodiments, multiple zones are alsoprovided in the vertical direction. For example, there may be 5 verticalzones per depth range, for a total of 125 resolution adjustment zones.An example of such zones extending in three dimensions is illustrated inFIG. 11B, and described below.

With continued reference to FIG. 11A1, a user's eyes 210, 220 fixate ona particular fixation point 1006 within the grid 1100. The displaysystem may determine the location of the fixation point 1006, and thezone in which the fixation point 1006 is located. The display system mayadjust resolutions of content based on the proximity of virtual contentto the fixation point 1006, which may include determining the proximityof the virtual content to the zone in which the fixation point 1006 islocated. As an example, for content included in a zone in which thefixation point 1006 is located, the resolution may be set at aparticular polygon count, which in the example is 10,000 polygons. Basedon a distance from the fixation point 1006, content included in theremaining zones may be adjusted accordingly. For example, contentincluded in an adjacent zone to a zone that includes the fixation point1006 may be rendered at a lower resolution (e.g., 1,000 polygons). Whilethe example of FIG. 11A1 illustrates adjusting a polygon count as anexample, as described herein, adjusting resolution may encompass makingother modifications to the resolution of presented content. For example,the adjustment in resolution may include one or more of: adjusting thepolygon count, adjusting primitives utilized to generate the virtualobject (e.g., adjusting a shape of the primitives, for example adjustingprimitives from triangle mesh to quadrilateral mesh, and so on),adjusting operations performed on the virtual object (e.g., shaderoperations), adjusting texture information, adjusting color resolutionor depth, adjusting a number of rendering cycles or a frame rate, andadjusting quality at one or more points within a graphics pipeline ofgraphics processing units (GPUs)).

In addition, while the example of FIG. 11A1 provides particular examplesof differences in polygon count in different resolution adjustmentzones, other absolute numbers of polygons and other rates of change inresolution with distance from the fixation point 1006 are contemplated.For example, while a drop-off of resolution from the fixation point 1006may be based on a drop-off rate symmetric about depth and lateraldistance from the fixation point 1006, other drop-off relationships mayalso be utilized. For instance, a lateral distance from the fixationpoint 1006 may be associated with a greater drop-off in resolutionrelative to a depth distance from the fixation point 1006. Furthermore,the size of each zone (e.g., size of a volume of space of the zone)included in the grid may optionally be different (e.g., the zones mayvary radially from a foveal axis). In some embodiments, the drop-off maybe continuous from the fixation point 1006, such that discrete zoneshaving assigned resolutions or resolution relationships with the zonecontaining the fixation point 1006 are not utilized. For instance, adrop-off from the fixation point 1006 to a particular zone 1108 (e.g., azone in which content is rendered at a resolution of 100 polygons) maybe modified to be a continuous drop-off from the fixation point 1006 toan edge of the grid (e.g., edge of the particular zone 1108). Asdescribed in further detail below with reference to FIGS. 54-59 , insome embodiments, such a drop-off in resolution may be associated with a“rolloff” attribute of a resolution distribution. It will be appreciatedthat each of the considerations above also apply to zones extending inthe vertical direction.

In some embodiments, the number and sizes of zones included in the gridmay be based on a confidence associated with a determination of theuser's fixation point 1006. For instance, the confidence may be based onan amount of time that the user's eyes have been fixed on the fixationpoint 1006, with a lesser amount of time being associated with a lesserconfidence. For example, the display system may monitor the user's eyeat a particular sampling rate (e.g., 30 Hz, 60 Hz, 120 Hz, 1 kHz), andmay increase a confidence in the fixation point 1006 as successivesamples indicate the user is generally maintaining the fixation point1006. Optionally, particular thresholds of fixation may be utilized, forinstance a fixation for a particular duration (e.g., 100-300milliseconds) on a same, or similar, fixation point may be associatedwith a high confidence, while less than the particular duration may beassociated with a lesser confidence. Similarly, fluctuations in theeyes, such as pupil dilation, and so on, which may affect adetermination of the user's fixation point, may cause the display systemto reduce the confidence. It will be appreciated that the display systemmay monitor the eye with sensors, such as camera imaging devices (e.g.,camera assembly 630, FIG. 6 ). Optionally, the display system mayutilize a combination of the sensors to determine an eye gaze of theuser (e.g., different eye gaze determination processes may be utilized,such as an infrared sensor utilized to detect infrared reflections fromthe eye and to identify a pupil, a visible light imaging device utilizedto detect an iris of the eye, and so on). The display system mayincrease a confidence when multiple eye gaze determination processes arein conformance, and may decrease the confidence level if they disagree.Similarly, for display systems which conduct only one of the eye gazedetermination processes, each eye gaze determination process may beassociated with a particular confidence level (e.g., one determinationprocess may be considered more accurate than others) and the sizes ofthe resolution adjustment zones may be selected, at least in part, onthe process being implemented.

In some embodiments, the display system may increase, or decrease, anumber of zones for each updating of the fixation point 1006. Forexample, more zones may be utilized as the confidence associated withthe fixation point 1006 increases and fewer zones may be utilized asconfidence decreases. FIG. 11A2 illustrates examples of representationsof top-down views of resolution adjustment zones at different times asthe sizes and numbers of the zones change. At time t=1, as seen in a topdown view, the user's field of view may be divided into an initial setof zones. At time t=2, confidence in the location of the fixation point1006 increases and the display system may also decrease the size of thezone that is occupied by the fixation point 1006 and that is render athigh resolution. Optionally, as illustrated, the sizes of the otherzones may also decrease. At time t=3, confidence in the location of thefixation point 1006 decreases and the display system may also increasethe size of the zone that is occupied by the fixation point 1006 andthat is render at high resolution. Optionally, as illustrated, the sizesof the other zones may also increase. It will be appreciated that aplurality of zones may also extend in the y-axis and that similarincrease or decreases in the sizes and numbers of zones may also beinstituted on that axis. For example, the sizes of the zones extendingvertically on the y-axis may decrease with increasing confidence, whilethe sizes may increase with decreasing confidence. Optionally, thedisplay system may determine a confidence of the fixation point 1006 foreach frame presented by the display system to the user and t=1, t=2, andt=3 may represent different frames. Since assigning more zones mayrequire an increase in computational power (e.g., the display system mayhave to adjust resolutions of more content, identify which zones contentare included in, and so on), the display system may balance the increasein required computational power afforded by the increase in the numberzones with the savings in computation power afforded by the potentialdecrease in the resolution of content.

With reference again to FIG. 11A1, the grid may change dynamically inthe sense that the fixation point 1006 may be set as being located at acenter (e.g., centroid) of the grid. Therefore, the display system mayavoid edge cases in which the fixation point 1006 is determined to belocated on vertices of the grid. For example, as the user's eyes rotateand then fixate on different three-dimensional locations in space, thegrid may be similarly moved with the user's gaze.

FIGS. 11B-11E illustrate examples of various resolution adjustment zoneconfigurations. Additional shapes and configures of resolutionadjustment zones that are not illustrated may be utilized, and theexamples should not be considered exhaustive. In addition, in somedrawings, the user's eyes 210, 220 may be illustrated spaced apart fromthe various resolution adjustment zones for ease and clarity ofillustration. For all these drawings, it will be appreciated that theeyes 210, 220 may be disposed at the boundary of, or in, the zone (see,e.g., FIG. 11A1).

FIG. 11B illustrates an example of a three-dimensional representation ofa portion of the resolution adjustment zones of FIG. 11A1. It will beappreciated that FIG. 11A1 may be understood to illustrate across-sectional view taken along the plane 11A1-11A1 of thethree-dimensional representation of FIG. 11B, with FIG. 11B omittingsome of the resolution adjustment zones of FIG. 11A1 for clarity ofillustration. With continued reference to FIG. 11A1, a field of viewprovided by a display system is separated into 27 zones. That is, thefield of view is separated into 3 depth ranges 1102B-1102D, and at eachdepth range a 3×3 grid of zones is included that extends laterally andvertically at the depth range.

A determined fixation point 1006 is illustrated as being within a zonelocated in the center of the field of view. Virtual objects locatedwithin zones outside of a zone that includes the fixation point 1006 maybe reduced in resolution according to a distance from the fixationpoint's 1006 zone, as discussed herein. Since the zones extend laterallyas well as vertically, reduction in resolution can occur based ondistance on lateral, vertical, and depth axes (x, y, and z-axesrespectively) from the resolution adjustment zone of the fixation point.For example, in some embodiments, virtual objects located in zone 1108can be reduced in resolution according to lateral distance as shown inFIG. 11A1 (e.g., zone 1108 includes a same vertical portion of theuser's field of view as the zone that includes the fixation point 1006,and may be on the same depth plane).

Similar to the above, and similar to the zones described in FIGS.110-11E below, the user's fixation point can optionally maintainedlocated at the center (e.g., centroid) of the zones, or the zones can befixed with respect to the user's field of view and the user's fixationpoint can be located within any of the zones.

FIG. 11C illustrates another example of a configuration for resolutionadjustment zones. In the example, a field of view provided by a displaysystem is illustrated as being separated into zones of ellipses thateach encompass a particular three-dimensional volume of space. Similarto FIG. 11A1, each zone (e.g., zone 1112A-112D) extends along lateraland depth dimensions. In some embodiments, each zone also extends toencompass at least a portion of the user's vertical field of view. Afixation point 1006 is illustrated as being at a center of the zones(e.g., within zone 1112A). Virtual objects located within zones outsideof zone 1112A may be reduced in resolution according to a distance fromzone 1112A, for instance according to the techniques described herein.For example, each zone outside of zone 1112A can be assigned aparticular resolution, or a drop-off can be utilized, to determine areduction in resolution. Zone 1112D is illustrated as being a furthestzone from zone 1110A, and the reduction in resolution can be thegreatest in zone 1112D.

FIG. 11C illustrates an example of a three-dimensional representation ofthe resolution adjustment zones of FIG. 11C, with FIG. 11C showing across-sectional view taken along the plane 11C-11C. In this example, thefield of view provided by the display system is illustrated as beingseparated into zones of ellipsoids that each encompass athree-dimensional volume of space. The user's fixation point 1006 isillustrated at a centroid of the user's field of view, and locatedwithin zone 1112A. Optionally, FIG. 11D can represent each ellipse ofFIG. 11C being converted into an ellipsoid. In some embodiments, thesize of FIG. 11C's zone 1112A along depth and lateral directions candefine the size of the principal axes of FIG. 11D's zone 1112A along theX and Z axes. The various zones may form concentric spheres orellipsoids.

FIG. 11E illustrates another example of a three-dimensionalrepresentation of the resolution adjustment zones of FIG. 11C, with FIG.11C showing a cross-sectional view taken along the plane 11C-11C. Thefield of view provided by the display system is illustrated as beingseparated into stacked levels of similar concentric zones. For example,FIG. 11E may represent the ellipses of FIG. 11C being extended along avertical direction to create cylinders. The cylinders may then beseparated in the vertical direction, such that each cylinder encompassesa portion of the user's vertical field of view. Therefore, FIG. 11Eillustrates 9 zones of cylinders. Each zone additionally excludes anyinterior zones (e.g., ellipsoid 1112B would encompass a volume of spacethat excludes a volume of space encompassed by ellipsoid 1112A). In theexample, the fixation point 1006 is illustrated as being within a centerzone 1110A, and virtual objects located outside of the center zone 1110Acan be reduced in resolution according to the techniques describedherein.

FIG. 12A illustrates a flowchart of an example process 1200 foradjusting resolutions of content according to proximity to athree-dimensional fixation point. For convenience, the process 1200 maybe described as being performed by a display system (e.g., the wearabledisplay system 60, which may include processing hardware and software,and optionally may provide information to an outside system of one ormore computers or other processing, for instance to offload processingto the outside system, and receive information from the outside system).

At block 1202, the display system determines a three-dimensionalfixation point of a user. As described above, the display system mayinclude sensors to monitor information associated with the user's eyes(e.g., the orientation of the eyes). A non-exhaustive list of sensorsincludes infrared sensors, ultraviolet sensors, visible wavelength lightsensors. The sensors may optionally output infrared, ultraviolet, and/orvisible light onto the user's eyes, and determine reflections of theoutputted light from the user's eyes. As an example, infrared light maybe output by an infrared light emitter, and an infrared light sensor. Itwill be appreciated that the sensor, which may include a light emitter,may correspond to the imaging device 630 of FIG. 6 .

The display system may utilize the sensors to determine a gazeassociated with each eye (e.g., a vector extending from the user's eye,such as extending from the fovea through the lens of the eye), and anintersection of the gazes of each eye. For example, the display systemmay output infrared light on the user's eyes, and reflections from theeye (e.g., corneal reflections) may be monitored. A vector between apupil center of an eye (e.g., the display system may determine acentroid of the pupil, for instance through infrared imaging) and thereflections from the eye may be used to determine the gaze of the eye.The intersection of the gazes may be determined and assigned as thethree-dimensional fixation point. The fixation point may thereforeindicate a location at which content is to be rendered at a full ormaximum resolution. For example, based on the determined gazes thedisplay system may triangulate a three-dimensional location in space atwhich the user is fixating. Optionally, the display system may utilizeorientation information associated with the display system (e.g.,information describing an orientation of the display system inthree-dimensional space) when determining the fixation point.

At block 1204, the display system obtains location informationassociated with content being, or that is to be, presented by thedisplay system to the user. Prior to rendering content for presentationto the user (e.g., via outputs of waveguides, as described above), thedisplay system may obtain location information associated with contentthat is to be presented to the user. For instance, as described above,the virtual content may be presented to the user such that the contentappears to be located in the real-world (e.g., the content may belocated at different depths within the user's field of view). It will beappreciated that the display system include or may have access to athree-dimensional map of the ambient environment, which can informlocations of any virtual content in this ambient environment. Withreference to this map, the display system may access and provideinformation specifying three-dimensional locations of virtual contentwithin the user's field of view (e.g., locations within a displayfrustum, as illustrated in FIGS. 10A-10B).

At block 1206, the display system adjusts resolution of virtual contentto be displayed to the user. The display system adjusts the resolutionof content based on its proximity to the three-dimensional fixationpoint. For instance, a rendering engine, such as a rendering engineimplemented by processing devices (e.g., central processing units,graphics processing units) which renders content for presentation to theuser, may adjust resources invested in rendering the content (e.g., therendering engine may adjust a resolution of the content).

The display system may determine a distance in three-dimensional spacebetween content to be presented to the user and the user's fixationpoint, and may reduce a resolution of the content based on thedetermined distance. The reduction may be determined according to adrop-off rate, for instance a continuous function that correlatesdistance to the resolution of content, and the display system may obtainthe resolution to render the content based on the continuous function.Optionally, the display system may determine the distance from acentroid of the content to the fixation point, and may render thecontent at a resolution based on the distance. Optionally, the displaysystem may render portions of a same content at different resolutionsaccording to the distance of various portions to the fixation point(e.g., the display system may separate the content into portions, andmay render further portions at reduced resolutions as compared to closerportions).

In some embodiments, the display system may access information usable toseparate a field of view of the user (e.g., corresponding to the displayfrustum) into zones, with each zone representing a volume of space inwhich content may be included. The accessed information, for example thegrid illustrated in FIG. 11A1, may indicate a particular resolution toutilize when rendering content that is to be included in each zone, withthe three-dimensional fixation point being set at a center of the grid.Additionally, the grid may indicate drop-offs in resolution to utilizewhen rendering content. For content that is included in multiple zones(e.g., content located in three-dimensional space claimed by two zones),the display system may optionally adjust a resolution of the content tocorrespond to a single zone, or optionally adjust portions of thecontent according to corresponding zones in which the portions arelocated.

When setting the resolution of content, the display system renderscontent located at the fixation point (e.g., in a same zone as thefixation point) at a full or maximum resolution. The maximum resolutionmay be based on a maximum value that hardware and/or software of thedisplay system is capable of rendering, while ensuring that content ispresented to the user at greater than a threshold refresh rate (e.g., 60Hz, 120 Hz) and optionally ensuring that the content is updated atspeeds greater than vergence rates (e.g., greater than 60 ms) andgreater than accommodation times (e.g., 20 ms to 100 ms) to reduce theperceptibility of changes in resolution. The display system maydynamically modify the maximum resolution, for instance prior to thedisplay system rendering each frame, based on available resources of thedisplay system. For example, as more content is to be presented to theuser, a maximum resolution of content may be decreased, ensuring thatthe display system may present frames of rendered content at abovethreshold rates desired for reducing the perceptibility of changes inresolution. The display system may optionally monitor the frames persecond at which content is being presented, and may adjust the maximumresolution, and/or adjust resolution drop-off rates based on distancefrom the fixation point, to ensure the presented frames per second doesnot drop below the threshold rate. As an example, the display system mayrender content, such as a first virtual object, located in the fixationpoint's zone at a maximum resolution. Instead of reducing the maximumresolution of the first virtual object, to ensure the frames per secondremains above a particular threshold, the display system may dynamicallyincrease drop-off rates of resolution based on distance. In this way,the display system may adjust resolutions assigned to each zone outsideof the fixation point's zone. Optionally, the display system may set aminimum resolution that may be used in each zone outside of the fixationpoint's zone, and may adjust the maximum resolution if the minimumresolution would be exceeded (e.g., if the display system needs toreduce resolution of content below the minimum to maintain the thresholdrate, the display system may reduce the maximum resolution). Similarly,the display system may reduce the maximum resolution while not reducingresolutions of content in zones outside of the fixation point's zone.Optionally, a user of the display system may indicate whether he/sheprefers that content located proximate to the fixation point is to begiven preference over other content.

In some embodiments, and as will be described in more detail below withrespect to FIGS. 13-14 , the display system may optionally utilize anangular proximity of content to a gaze of the user to adjust resolutionof the content. For example, if particular content is located outside ofa zone in which the fixation point is located, but is within a thresholdproximity of a gaze of the user such that the particular content willfall on a fovea of the user's eye, the display system may cause theparticular content to be rendered at a greater resolution (e.g., themaximum resolution, or at a resolution greater than indicated in thegrid illustrated in FIG. 11A1). Optionally, the display system mayreduce a resolution of the particular content, and apply a blurringprocess (e.g., Gaussian blur) to the particular content. In this way,the particular content may be rendered at a lesser resolution, whilebeing blurred to represent that the particular content is, for instance,further away from the user than the fixation point. In addition, theblurring may reduce the perceptibility of the lower resolution (e.g.,the blurring may reduce the perceptibility of increases in pixel sizedue to the lower resolution).

Example operations associated with presenting virtual content areillustrated in FIGS. 12B-12C (e.g., a rendering pipeline). In theexample of FIG. 12B, a three-dimensional scene is presented to a user,without adjustments to resolution made as described herein. In FIG. 12C,adjustments to resolution are performed according to fixation pointinformation as described herein. For example, one or more of thefollowing adjustments can be performed: reducing vertex operationcomplexity, reducing tessellation level of detail, reducing geometrygeneration, reducing pixel operation complexity/aggregation of multiplepixels, and so on. The adjustments, as illustrated, can advantageouslybe performed at different steps within a pipeline to present virtualcontent, and can be optimized according to particular software and/orhardware utilized to present the virtual content. It will be appreciatedthat the fidelity zones noted in FIG. 12C are resolution adjustmentzones.

With reference again to FIG. 12A, the display system presents adjustedcontent to the user at block 1208. As described above, the displaysystem has adjusted the resolutions of content based on proximity to thethree-dimensional fixation point. Subsequently, the display systempresents rendered content at associated locations to the user. In someembodiments, the display system may perform process 1200 for each frameof content to be rendered, or may adjust resolutions of content as theuser adjusts his/her fixation point.

As noted above, in some embodiments, virtual objects may be within auser's line of sight while also being presented at different depths.FIG. 13 illustrates an example of a representation of a user viewingmultiple virtual objects aligned with the user's line of sight. Theexample representation includes a user's field of view (e.g., displayfrustum 1004 of the display system), along with a gaze 1003A, 1003B ofthe user's eyes 210, 220, which are fixated at a fixation point on afirst virtual object 1008A.

As illustrated, a second virtual object 1008B is within an angularproximity of a gaze of the user (e.g., one or both of gaze vectors1003A, 1003B) such that the second virtual object 1008B will fall on theuser's fovea (e.g., fall on at least one fovea of either eye). Forexample, upon rendering frame 1110, the second virtual object 1008B islocated behind (e.g., at a greater perceived depth from) the firstvirtual object 1008A. It will be appreciated that the fovea is theportion of the retina having the highest visual acuity. Since the secondvirtual object 1008B will fall on the user's fovea, if a resolution ofthe second virtual object 1008B is reduced (e.g., reduced as describedabove, with respect to, at least, FIG. 11A1) the user may perceive thereduction in resolution. To avoid a perceptible reduction in resolution,the display system may (1) cause the second virtual object 1008B to berendered at a same resolution as the first virtual object 1008A, orwithin a threshold resolution of the first virtual object 1008A, and/or(2) cause the second virtual object 1008B to be rendered at a reducedresolution (e.g., as indicated in FIG. 11A1) and apply a blur to thesecond virtual object prior to presentation to the user. Without beinglimited by theory, the blur may mask the reduction in resolution whileproviding a depth cue.

FIG. 14 is a flowchart of an example of a process 1400 for adjustingvirtual content based on angular distance from a user's gaze. Forconvenience, the process 1400 will be described as being performed by adisplay system (e.g., the wearable display system 60, which may includeprocessing hardware and software, and optionally may provide informationto an outside system of one or more computers or other processing units,for instance to offload processing to the outside system, and receiveinformation from the outside system). In the example process 1400, thedisplay system is a vari-focal display system, in which each frame ispresented on the same depth plane, and optionally having all content tobe presented collapsed into a single frame buffer; that is, thevari-focal display system presents virtual content on one depth plane ata time.

The display system determines a three-dimensional fixation point of auser (block 1402) and obtains location information associated withpresented content (block 1404). The blocks 1402 and 1404 may correspondto the blocks 1202 and 1204, respectively, of FIG. 12A. As describedabove with reference to FIG. 12A, the display system monitors eyemovements (e.g., eye orientations) of the user and determines fixationpoints of the user. The display system may obtain location informationof content to be presented (e.g., in a next frame), and may subsequentlyadjust resolutions of the content.

With continued reference to FIG. 14 , the display system determinescontent to be reduced in resolution and that is located within athreshold angular distance from the user's gaze (block 1406). Thedisplay system identifies content that is to be reduced in resolutiondue to the proximity of the content from the fixation point (e.g., thecontent is located at a greater depth than the fixation point), but thatwill fall on the user's fovea (e.g., fall within a threshold angle fromthe user's gaze). Since the content will fall on the user's fovea, theuser may be able to perceive the reduction in resolution, as by thethree-dimensional fixation point foveated rendering described herein. Itwill be appreciated that content block 1406 may comprise performing theblocks illustrated in FIG. 12C, particularly the blocks identified inthe section “GPU”.

Consequently, at block 1408, the display system may optionally cause thedetermined content to be rendered at a greater resolution. The displaysystem may adjust the resolution of the determined content to be at fullresolution (e.g., at the same resolution as content located at thefixation point, or within a same zone, or volume of space, as thefixation point), or to be at greater than the reduced resolution thatwould otherwise be assigned to the content (e.g., as described in block1406).

At block 1410, display system may optionally reduce the resolution ofthe content, and may blur the content prior to presentation to the user.As described above, a vari-focal display system may utilize a singledisplay buffer to present content to the user. Since the vari-focaldisplay system is presenting all content at the same depth plane, thevari-focal display system may utilize the same display buffer to outputthe content, for instance, from a rendering engine.

Optionally, the display system may utilize initial depth buffers, witheach depth buffer assigned one or more depth planes, and may combine theinitial depth buffers to obtain the display buffer. With reference tothe illustration of FIG. 13 , a first depth buffer may include the firstvirtual object 1306, while a second depth buffer may include the secondvirtual object 1308. The display system may then apply a blurringprocess to the second depth buffer, or to particular content included inthe second depth buffer (e.g., the display system may apply the blurringprocess to the second virtual content 1308, but not to other contentlocated on a same depth plane but at a further angular distance from theuser's gaze). After performing the blurring process, the display systemmay combine the first depth buffer and second depth buffer (e.g., thedisplay system may add occlusions, for instance removing a portion ofthe second virtual object 1308 not visible due to occlusion by the firstvirtual object 1306), to obtain the display buffer.

An example blurring process may include the display system performing aconvolution of a kernel associated with blurring (e.g., a Gaussiankernel, circular kernel such as to reproduce a bokeh effect, box blur,and so on) to the content. In this way, the reduction in resolution maybe masked, while the processing savings from reducing the resolution maybe maintained. Optionally, a strength associated with the blurringprocess (e.g., a degree to which the content is blurred) may be based ona difference in depth between the user's fixation point and the content,and/or an angular proximity of the content to the user's gaze. Forexample, the degree of blurring may increase with increasing proximityto the user's gaze.

In some embodiments, the display system may utilize the features ofblock 1408 or 1410 according to hardware and/or software of the displaysystem. For example, particular hardware (e.g., graphics processingunits) may be able to perform the blurring process in hardware without athreshold hit to performance of the hardware. For this particularhardware, the display system may be configured to reduce resolution ofcontent and then blur the content. However, other hardware may be slowto perform the blurring process, and rendering content at greaterresolutions might enable greater performance. For this other hardware,the display system may be configured to render content at greaterresolutions. Furthermore, the decision between whether to render contentat a greater resolution, or at a lower resolution with blurring maydepend on the type of content to be displayed. For instance, the displaysystem may be configured to render text at a greater resolution, whilerendering shapes at a lower resolution and blurring.

With continued reference to FIG. 14 , at block 1412 the display systempresents content to the user. The display system may present theadjusted content to the user, for instance from a same display buffer asdescribed above.

II. Changes in Resolution Based on Ambient Illumination Levels

In addition to or as an alternative to reductions in resolution alongthe z-axis, various other schemes for presenting virtual content withreductions in resolution may be implemented in some embodiments.Advantageously, as noted herein, some aspects of the virtual content maybe presented at relatively high resolution and some other aspects may bepresented in relatively low resolution, which may reduce the use ofcomputational and energy resources by the display system, whilepreferably having low impact on the perceived image quality of thevirtual content.

With reference now to FIG. 15 , an example is illustrated of arepresentation of the retina of an eye of a user. The illustrated viewshows a retina 1500 as seen when viewed head-on along the visual axis ofthat retina. The retina 1500 includes a fovea 1510 surrounded by aperipheral area 1530. Within the fovea 1510 is the foveola 1520, whichintersects the visual axis.

It will be appreciated that the retina includes two types ofphotoreceptors: rods and cones. In addition, the distributions of thesephotoreceptors across the retina varies, providing different rod andcone densities across the retina.

With reference now to FIG. 16 , an example of resolution, and rod andcone density, across the retina 1500 of FIG. 15 is graphicallyillustrated. The x-axis indicates degrees of eccentricity relative to apoint at which the visual axis intersects the retina. The rightwarddirection on the page is the nasal direction and the leftward directionon the page is the temporal direction. As illustrated, the resolution ofthe human eye roughly correlates with the densities of photoreceptors(rods and cones) in the retina. Consequently, in some embodiments, thereduction or taper in the resolution (e.g., spatial resolution) ofvirtual content on the x and y-axes (e.g., on a given depth plane) maysubstantially follow the reductions across the retina of cone density,rod density, or an aggregate of rod and cone density. For example, thetrend of the resolution reduction away from the fixation point acrossthe user's field of view may be within ±50%, ±30%, ±20%, or ±10% of thetrend in the changes in the photoreceptor density (e.g., cone density,rod density, or an aggregate of rod and cone density) over correspondingportions of the retina. In some embodiments, the reduction in resolutionaway from the fixation point is gradual and substantially follows thedensity changes. In some other embodiments, the reduction in resolutionmay occur in steps (e.g., one step, two steps, etc.). For example, theremay be two steps: a highest resolution region of the field of viewcorrelated with the foveola, a medium resolution region correlated withthe fovea, and a lower resolution region correlated with the peripheralarea.

With continued reference to FIG. 16 , it will be appreciated thatdifferent photoreceptors have different levels of activity underdifferent light conditions, e.g., at different ambient illuminationlevels. As a result, it is possible that, while reductions in resolutionthat follow the densities of photoreceptors may not be consciouslyperceptible to the user at some illumination levels, they may beperceptible at other illumination levels. Consequently, in someembodiments, reductions in the resolution of virtual content, along thex, y, or z-axes, maybe set with reference to external light conditions.

For example, the vision behavior of the eye may be divided into threemodes, based on the light conditions. The three modes are photopicvision, mesotopic vision, and scotopic vision. Photopic vision typicallyoccurs in bright conditions, e.g., ambient light or illumination levelsof about 3 cd/m² or more, including about 10 to 10⁸ cd/m². In photopicvision, cones are primarily active. In scotopic vision, rods areprimarily active. In mesotopic vision, both rods and cones may beactive. As used herein, ambient light conditions or illumination levelsrefer to the amount of light that the eye of the user and his/her retinaare exposed to.

Mesotopic vision typically occurs under lower light conditions, e.g.,illumination levels of about 10⁻³ to 10^(0.5) cd/m². Both cones and rodsare active in at least some illumination levels within mesotopic vision,with the dominance of the rods or cones changing over time dependingupon whether ambient illumination levels are increasing or decreasing.As the eye adapts to a brighter environment, more cones become activatedin comparison to rods; on the other hand, as the eyes adapt to a darkenvironment, more rods are activated in comparison to cones.

Scotopic vision typically occurs in light conditions in which theillumination levels are less than the illumination levels for photopicvision. For example, scotopic vision may occur at illumination levels ofabout 10⁻² cd/m² or less, or about 10⁻³ cd/m² or less, including about10⁻³ to 10⁻⁶ cd/m². Rods are primarily active in scotopic vision. Itwill be appreciated that the illumination levels noted herein forphotopic, mesotopic, and scotopic vision are examples. In someembodiments, the illumination levels associated with each of type ofvision may be assigned arbitrarily, based on user preferences, and/orcustomization for a group to which the user belongs (e.g., based ongender, age, ethnicity, the presence of visual abnormalities, etc.).

In some embodiments, the type of vision (photopic, mesotopic, orscotopic) active in the user may be determined based on measurements ofambient illumination levels. For example, the display system may beconfigured to measure ambient illumination levels using a light sensor,such as the outwardly-facing camera 112 (FIG. 9D). In some embodiments,the display system may be in communication with another sensor or devicewhich provides information regarding the ambient illumination levels.

It will be appreciated that head-mounted display systems may block orattenuate some of the ambient light, such that an outwardly-facingcamera may not give luminance levels that accurately reflect the amountof light impinging on the eye. In addition, the display system, inprojecting light to the eye to provide virtual content, is also a sourceof light that may alter the illumination levels to which the eye isexposed. In some other embodiments, an inwardly-facing camera may beutilized to determine luminance levels. For example, luminance levelsare roughly correlated with the size of the pupil. FIG. 17 graphicallyillustrates an example of the relationship between pupil size and theamount of light incident on an eye of a user. The x-axis shows valuesfor luminance and the y-axis shows values for pupil area. Consequently,the display system may be configured to determine the pupil area of theuser and then extrapolate luminance based on this pupil area. Forexample, the display system may be configured to use the inwardly-facingcamera 500 (FIG. 6 ) to capture an image of the eye 210 of the user andthen analyze the image to determine the pupil area or other metricindicative of pupil area (e.g., pupil diameter or width). For example,the area occupied by the pupil of the eye 210 in the image captured bythe camera may be determined and then corrected for any scaling factorcaused by the optics of the camera. Advantageously, using pupil area todetermine luminance levels may effectively take into account bothreductions in ambient luminance levels caused by the display blockingsome ambient light and also contributions to the luminance levels by thelight output of the display itself.

With continued reference to FIG. 17 , the display system may beconfigured to determine whether the user's eyes are in a photopic,mesotopic, or scotopic vision mode based upon the determined pupil area.For example, the display system may have resident in memory a table orother stored information specifying the vision mode expected forparticular pupil area. As examples, in line with the graph shown in FIG.17 , the display system may categorize pupil areas of about 3 mm² orless as being indicative of photopic vision, pupil areas of 3 mm² ormore up to about 38 mm² as being indicative of mesotopic vision, andpupil areas of more than 38 mm² as being indicative of scotopic vision.It will be appreciated that these luminance values and associated visionmodes are examples and that other values may be substituted. Forexample, different values may be applied to different users in responseto input from the users, or different values may be applied based on theparticular category in which the user may fall (e.g., gender, age,ethnicity, the presence of visual abnormalities, etc.). In addition, itwill be appreciated that the display system does not necessarilyidentify a specific vision mode. Rather, the display system may beconfigured to simply associate particular measured pupil areas withparticular resolution levels or adjustments.

In some embodiments, inputs from both the inwardly-facing camera 510(FIG. 6 ) and the outwardly-facing camera 112 (FIG. 9D) may be utilizedto determine luminance levels. For example, the display system may beconfigured to take an average (including a weighted average) of theluminance levels determined using the cameras 510 and 112. As notedabove, the luminance level determined using the camera 510 may beextrapolated from the size of the pupil area of the user's eye, based onimaging the user's eye using that camera 510.

It will be appreciated that rods and cones have different levels ofvisual acuity and different sensitivities to color and contrast.Consequently, because ambient luminance levels impact whether rodsand/or cones are active, there are differences in visual acuity andsensitivities to color and contrast at different ambient luminancelevels. Advantageously, the light-level differences in visual acuity andsensitivities to color and contrast may be applied to provide additionalbases for reducing resolution, which may be utilized in conjunction withchanges in resolution based on the fixation point as described above(e.g., regarding FIGS. 12A and 14 ), or may be utilized separately evenwithout specifically making changes in resolution based on the fixationpoint.

With reference now to FIG. 18 , a diagram is shown of an example of aprocess 1800 for adjusting virtual content based on the amount of lightincident on an eye of a user. For convenience, the process may bedescribed as being performed by a display system (e.g., the wearabledisplay system 60 (FIG. 9D), which may include processing hardware andsoftware, and optionally may provide information to an outside system ofone or more computers or other processing units, for instance to offloadprocessing to the outside system, and receive information from theoutside system).

At block 1810, the display system determines the amount of lightreaching the retina. Preferably, this determination is an estimate ofthe amount of light reaching the retina rather than a direct measurementof light that impinges on the retina. This estimate may be made asdiscussed herein using the methods disclosed for determining luminancelevels. For example, luminance levels may be assumed to correspond tothe amount of light reaching the retina. As result, determining theamount light reaching the retina may include determining a size of theuser's pupil and/or determining ambient luminance levels using a sensorconfigured to detect light, such as an outwardly-facing camera on adisplay device.

At block 1820, the display system adjusts the resolution of virtualcontent to be presented to the user based on the amount of light foundto be reaching the retina at block 1810. In some embodiments, adjustingthe resolution of the virtual content comprises adjusting one or more ofthe spatial resolution, color depth, and light intensity resolution ofthe virtual content. It will be appreciated that the human visual systemhas the greatest acuity and sensitivity to spatial resolution, color,and light intensity under photopic illumination levels. The ability toperceive differences in spatial resolution, color, and light intensitydecrease under mesotopic illumination levels, and further decrease underscotopic illumination levels.

Consequently, in some embodiments, if the amount of light present isfound to correspond to the levels for photopic vision, then virtualobjects may be rendered at full or high spatial resolution (compared tospatial resolution which would be utilized for mesotopic or scotopicvision). If the amount of light present is found to correspond tomesotopic levels, then virtual objects may be rendered at may reducespatial resolution compared to the spatial resolution utilized forvirtual objects under photopic illumination levels. If the amount oflight is found to correspond to scotopic levels, then the virtualobjects may be rendered at a spatial resolution that is lower than thatused under mesotopic or photopic illumination levels. Spatial resolutionmay be adjusted as described herein, e.g., by reducing the number ofpolygons, etc.

Color depth or bit depth may similarly be adjusted depending onillumination levels, with the highest color depth used under photopicillumination levels, an intermediate color depth used under mesotopicillumination levels, and the lowest color depth used under scotopicillumination levels. It will be appreciated that color depth may beadjusted by changing the number of bits used for each color component ofa pixel, with fewer bits equating to lower color depth.

Likewise, without being limited by theory, gradations in light intensityare believed to become larger as illumination levels progress fromphotopic to mesotopic to scotopic illumination levels. Stated anotherway, the human visual system is believed to be able to discern fewerdifferences in light intensity as the ambient illumination leveldecreases. In some embodiments, the display system may be configured todisplay fewer gradations in light intensity as illumination levelsprogress from photopic to mesotopic to scotopic illumination levels. Asa result, the largest number of gradations in light intensity levels arepresented under photopic illumination levels, fewer gradations arepresented under mesotopic illumination levels, and yet fewer gradationsare presented under scotopic illumination levels.

In addition, in some embodiments, the display system may be able toprovide a larger number of gradations in light intensity than the useris able to perceive. An example of this illustrated in FIGS. 22 a-22 c ,discussed further below. For example, the display system may be able todisplay 256 different levels of intensity for a given image pixel, butthe user may only be able to perceive a lower number of levels, e.g., 64levels. In this instance, multiple possible light intensity levels aresubsumed within a single one of the perceptible light intensity levels.For example, the display system may be able to display four differentlight intensity levels, but the user may perceive all four as beingsimilar. In such circumstances, where multiple possible lightintensities are perceived by the user as being the same, the displaysystem may be configured to select the lowest intensity value, out ofthese values that are perceived to be similar, for display. As a result,the display system may be able to utilize lower intensities, therebyreducing the amount of power used to illuminate a display to achieve thedesired light intensities. This may have particular advantages indisplay systems in which individual pixels of a spatial light modulatorare themselves light emitters, such as organic and inorganic LEDs. Insome embodiments, the number of gradations decrease with decreases inambient illumination levels, and the display system is configured togroup a larger number of possible light intensity levels together, todisplay the lowest light intensity of the group.

It will be appreciated that, for virtual content that is to bedisplayed, one, two, or all three of spatial resolution, color depth,and light intensity resolution may be changed based on the lightconditions to which a user is subjected (the amount of light reachingthe user's retina). These adjustments to spatial resolution, colordepth, and/or light intensity resolution based on light conditions maybe made to virtual content overall, without making adjustments toresolution based on distance from the fixation point of the user's eyes,as disclosed herein. In some other embodiments, the adjustments tospatial resolution, color depth, and/or light intensity resolution basedon light conditions may be made in conjunction with adjustments toresolution based on distance from the fixation point (see, e.g., FIGS.12A and 14 ). In some embodiments, if resolution decreases with distancefrom the fixation point, the profile of the decrease on a given plane(on the x and y-axes) preferably matches the profile of changes in conedensity across corresponding portions of the retina.

In some embodiments, as noted herein, adjustments to spatial resolution,color depth, and/or light intensity resolution are preferably tied tothe mode of vision (photopic, mesotopic, or scotopic vision) active at agiven time. These adjustments may dynamically change if the mode ofvision changes. For example, when the user progresses from photopicvision to scotopic vision, resolution may decrease as discussed herein.Conversely, when the user progresses from scotopic vision to mesotopicvision, the resolution of virtual content may increase. It will beappreciated that tying resolution adjustments to a particular mode ofvision does not require a specific determination that the user is inthat particular mode; rather, the display system may be configured tosimply associate particular ranges of ambient illumination levels orpupil size with particular resolutions, whether spatial resolution,color depth, or light intensity resolution. In addition, while theresolution adjustments are preferably tied to three levels of lightconditions (corresponding to three modes of vision) as discussed herein,in some embodiments, the resolution adjustments may be tied to twolevels of light conditions, or more than three levels of lightconditions.

It will also be appreciated that the resolution adjustment may occur inreal time (e.g., as ambient light conditions change), or may be delayedfor a set duration to allow the human visual system to adapt to existinglight conditions before the resolution adjustment to virtual content ismade. Without being limited by theory, it is believed that the humanvisual system requires a period of time to adapt to differentillumination levels, with that period of time increasing as illuminationlevels decrease. Consequently, in some embodiments, adjustments inresolution due to changing illumination levels are not made until theuser has been exposed (e.g., substantially continuously exposed) to aparticular illumination level for a set amount of time. For example, theset amount time may be 5 minutes, 10 minutes, 15 minutes, or 20 minutes.

With continued reference to FIG. 18 , at block 1830, virtual content ispresented to the user. The presentation of this virtual content may beconducted as discussed herein, e.g., as in block 1208 of FIG. 12A orblock 1412 of FIG. 14 .

With reference now to FIG. 19 , an example is graphically illustrated ofa change in resolution detectable by the eye of a user as the amount oflight incident on the eye changes. This figure illustrates an example ofthe sensitivity of the human visual system to spatial resolution underdifferent vision modes. Scotopic vision occurs in the low-light region1910, mesotopic vision occurs in the medium-light region 1920, andphotopic vision occurs in the bright light region 1930. As shown,sensitivity to spatial resolution decreases substantially as ambientillumination levels decrease. In some embodiments, the adjustments tospatial resolution discussed above regarding FIG. 18 correspond to thecontours of the illustrated curve. For example, for a given light levelin the photopic or scotopic vision mode, the virtual content is renderedwith sufficient spatial resolution to meet or exceed the resolutionvalues shown on the y-axis.

With reference now to FIG. 20 , it will be appreciated that differentphotoreceptors may be used to perceive light of different wavelengths orcolors. FIG. 20 graphically illustrates an example of differences insensitivity of the eye to light of different colors at different levelsof illumination. The differences in time duration on the x-axis arereflective of the amount of time typically needed for the human visualsystem to adapt to a particular ambient illumination level, such that aparticular mode of vision is activated. Notably, at ambient illuminationlevels corresponding to scotopic vision and a portion of mesotopicvision, photoreceptors for red light may no longer be active, whilephotoreceptors for blue light are active under the lowest lightconditions. It will be appreciated that red, green, and blue lightcorrespond to the colors most typically used as component colors in adisplay system to form full color images (e.g., as discussed hereinregarding FIG. 8-9B). In some embodiments, the display system may beconfigured to vary the rendering of images of different colors dependingupon the ambient illumination levels.

With reference now to FIG. 21 , a diagram is shown of an example of aprocess 2100 for adjusting virtual content formed using multiplecomponent color images, where the resolution adjustment is made based onthe color of the component color image. At block 2110, the displaysystem provides virtual content to be presented using multiple componentimages. These may be different images of different component colors tobe directed to different waveguides, as discussed regarding FIG. 8-9B.Consequently, in some embodiments, each of the streams of images ofdifferent component colors may be separately rendered. Providing virtualcontent to be presented using multiple component images may includeutilizing a display system that outputs image streams of differentcomponent colors to form a full color image.

At block 2120, the display system may adjust resolutions of componentcolor images based on their color. For example, the display system mayselect color images of one of these component colors for resolutionadjustment. For example, the selection may be made based on adetermination of illumination levels, as discussed above regarding block1810 of FIG. 18 . As shown in FIG. 19 , some component colors may not beperceived by a user at some illumination levels. The display system mayhave stored within it information regarding illumination levels andcomponent colors that are not visible at those levels. If there is amatch between the illumination level and the component color not visibleat those levels, then images of that component color may be selected foradjustment. In some environments, one adjustment may be to simply notrender or display that component color image if the ambient illuminationlevels are such that the user is not expected to perceive that color.For example, under scotopic illumination levels, the display system maybe configured to not render or display images of the component colorred.

With continued reference to FIG. 21 , at block 2130, virtual content ispresented to the user. The presentation of the virtual content may beconducted as discussed herein, e.g., as in block 1208 of FIG. 12A orblock 1412 of FIG. 14 .

With reference now to FIGS. 22A-22C, as discussed above and withoutbeing limited by theory, the ability of the human visual system toperceive gradations in light intensity is believed to change withambient illumination levels. FIGS. 22A-22C show examples of changingcontrast sensitivity as the amount of light incident on the eye of theuser decreases. For example, FIG. 22A may be understood to show thecontrast sensitivity under photopic light conditions, FIG. 22B may beunderstood to show the contrast sensitivity under mesotopic lightconditions, and FIG. 22C may be understood to show the contrastsensitivity under scotopic light conditions. FIG. 22A shows aprogression 2100 of gradations 2110 ₁ to 2110 _(i), proceeding from highlight intensity at the top to low with light intensity at the bottom.Similarly, FIG. 22B shows a progression 2102 of gradations 2110 ₁ to2110 _(i), proceeding from high light intensity to low with lightintensity. Likewise, FIG. 22C shows a progression 2104 of gradations2110 ₁ to 2110 _(i), proceeding from high light intensity to low lightintensity. The boxes 2120, 2130, 2140, indicate the groups of intensitygradations which are perceived by the user is being the same. The sizesof these groups are expected to increase with decreasing ambientillumination levels, as illustrated. Consequently, as discussed aboveregarding FIG. 18 , in some embodiments, the display system may beconfigured to use the lowest intensity value within each group (e.g.,within each of the boxes 2120, 2130, 2140).

With reference now to FIG. 23 , an example of a representation of theoptic nerve and peripheral blind spots of the eyes of a user isillustrated. In some embodiments, in addition to or as an alternative toany of the resolution adjustments disclosed herein, the display systemmay be configured to refrain from rendering content in various locationswhere content is not expected to be perceptible by the user. FIG. 23illustrates left and right eyes 210 _(L) and 210 _(R), respectively.Each eye has a respective optical axis 1003A and 1003B and optical nerve2300 _(L) and 2300 _(R). There is a blind spot of the point where eachof the optical nerves 2300 _(L) and 2300 _(R) contact their respectiveeyes 210 _(L) and 210 _(R). These blind spots prevent the viewer fromseeing content in the direction of the rays 2302 _(L) and 2302 _(R). Inaddition, at the periphery of each eye there exists a region in whichcontent cannot be seen by the opposite eye. For example, content in theleft peripheral region P_(L) may be seen by the left eye 210 _(L), butis not seen by the right eye 210 _(R). On the other hand, content in theright peripheral region P_(R) may be seen by the right eye 210 _(R), butis not seen by the left eye 210 _(L). Consequently, in some embodiments,the display system may be configured to omit rendering content thatwould be mapped to the blind spots of each eye 210 _(L) and 210 _(R),e.g., content falling on the rays 2302 _(L) and 2302 _(R). In additionor alternatively, in some embodiments, the display system may beconfigured to omit rendering content to the left eye 210 _(L) if thatcontent falls within the right peripheral region P_(L); and/or thedisplay system may be configured to omit rendering content to the righteye 210 _(R) if that content falls within the left peripheral regionP_(L). It will be appreciated that the locations of the blind spotsand/or the peripheral regions may be preset, e.g., based on averages fora population of users and/or may be tailored and calibrated for aparticular user by test using content displayed at various locations andinputs from the user indicating whether or not a virtual object isvisible.

Multiple Image Streams for Providing Content Having DifferentResolutions

In some embodiments, a foveated image having high and low spatialresolution regions may be formed by spatially overlapping two or moreimage streams, each having a different resolution (e.g., a differentperceived pixel density). For example, one of the image streams, e.g.,the low resolution image stream, may form images having a large field ofview and another of the image streams, e.g., the high-resolution imagestream, may form images having a narrow field of view. The narrow fieldof view image and the high field of view image may contain similarcontent, although at different resolutions or pixel densities as seen bythe user. These images may be overlaid one another (e.g., occupy thesame location in space simultaneously or in close temporal proximity,such that the viewer perceives the images are being presentsimultaneously). Thus, the viewer may receive an aggregate image havinghigh-resolution in a confined part of their field of view and lowresolution over a larger portion of their field of view. Preferably, asdiscussed herein, the high-resolution portion maps to the foveal visionregion of the user's eyes while the low resolution portion maps to theperipheral vision region of the user's eyes. As such, the differences inresolution between the high-resolution portion and the low resolutionportion of the image is preferably not readily perceptible to the user.

In some environments, the display system for displaying the high and lowresolution images utilizes the same spatial light modulator to form bothimages. Thus, the spatial light modulator has a fixed size and densityof pixels. In display systems with a fixed size and density of pixels,an increase in angular field of view (FOV) comes at the cost of spatialor angular resolution, e.g., as governed by the Lagrange invariant. Forexample, if an SLM having a fixed number of pixels is used to form boththe high and low resolution images, then spreading those pixels acrossthe entire field of view would provide an image with a lower apparentresolution than confining those pixels to a small portion of the totalfield of view; the pixel density of the high-resolution images is higherthan the pixel density of the low-resolution images. Consequently, thereis generally an inverse relationship between FOV and angular resolution.Because FOV and angular resolution affect image visibility and quality,this tradeoff places constraints on user experience and the ultimateachievable FOV and angular resolution in AR or VR systems. As will beapparent from the discussion herein, in some embodiments, the term“resolution” may be used to refer to “angular resolution.”

Head-mounted display devices or wearable display devices can beconfigured to provide an immersive user experience by projecting virtualcontent directly into the eyes of a user. Although it can be beneficialto provide wide FOV images at a uniformly high resolution across theFOV, the physiological limitations of the human visual system canprevent a user from appreciating or even noticing high resolutionimagery positioned in the peripheral regions of the user's field ofview. This inability to perceive high resolution imagery within theperipheral regions is caused by characteristics of the retina of a humaneye, which contains two types of photoreceptors, namely rod cells andcone cells. The cones are more responsible for acute (detailed) vision.The rods and cones are distributed differently in the human eye. Thehighest concentration of cone cells is found within the fovea (i.e., thecenter of the retina), while the highest concentration of rod cells isfound in the region immediately surrounding the fovea (i.e., theperiphery of the retina). Because of this non-uniform distributions ofthe rod cells and cone cells, the fovea is responsible for sharp centralvision (also called foveal vision). Visual acuity decreases as distancefrom the fovea increases.

For AR or VR applications, a headset is generally worn by one user at atime. The headset can be configured to take advantage of the user'sinability to perceive all the details of a wide field of view stream ofimages at once by limiting the display of high-resolution content toregions within the wide field of view currently being focused on by theuser. In this way, the headset can provide the user with the appearanceof a high-resolution wide FOV stream of images without the need for theprocessing power that would otherwise be required to generatehigh-resolution content across the entire field of view. The stream ofimages presented to the user can take many forms and will be generallyreferred to as an image stream. For example, the image stream can show astatic image by continuously displaying the same image to the user orcan show motion by displaying a stream of different images. In someembodiments, the headset can be configured to display more than oneimage stream at the same time; the different image streams can havedifferent angular resolutions and can extend across different regions ofthe user's FOV. It should be noted that an image stream associated withan AR system might not display content entirely across a particularregion to which it is assigned since AR systems are designed to mixvirtual content with real-world content.

According to some embodiments, a first image stream and a second imagestream can be presented to a user simultaneously, or in rapid successionsuch that the two image streams appear to be displayed simultaneously.The first image stream can have a wide FOV and low resolution that canencompass the user's vision to evoke an immersion experience to theuser. The second image stream can have a narrow FOV and a highresolution that can be dynamically displayed within the boundaries ofthe first image stream according to the user's current fixation point asdetermined in real-time using eye-gaze tracking techniques. In otherwords, the second image stream can be shifted around as the user's eyegaze changes, such that the second image stream persistently covers theuser's foveal vision. In some embodiments, the first image stream ispresented to the user at a fixed position, as the second image stream isshifted around relative to the first image stream. In some otherembodiments, both the first image stream and the second image stream areshifted according to the user's current fixation point.

The content of the second image stream can include a subset of thecontent of the first image stream with a higher resolution than thefirst image stream, and can be overlaid on and properly aligned withrespect to the first image stream. Because the higher resolution secondimage stream overlays the portion of the first image stream within theuser's foveal vision, the lower resolution of the first image stream maynot be perceived or noticed by the user. In some embodiments, the subsetof the content of the first image stream overlaid by the second imagestream can be turned off or be presented with a lower intensity for moreuniform brightness and for better resolution perception. In this way,the user can perceive the combination of the first image stream and thesecond image stream as having both a wide FOV and high resolution. Sucha display system can afford several advantages. For example, the displaysystem can provide a superior user experience whiling having arelatively small form factor and saving computing resources andcomputing power.

According to some embodiments, a first light beam associated with thefirst image stream and a second light beam associated with the secondimage stream can be multiplexed into a composite light beam usingcertain multiplexing methods. For example, time-division multiplexing,polarization-division multiplexing, wavelength-division multiplexing,and the like, can be used according to various embodiments. Thecomposite light beam can be directed to one or more optical elementsthat serve to de-multiplex the composite light beam into two separateoptical paths. For example, a beam splitter such as a polarization beamsplitter (PBS) or a dichroic beam splitter, or optical switchingelements can be used to separate the composite light beam depending onthe method of multiplexing used. Once separated, the first light beamassociated with the first image stream and the second light beamassociated with the second image stream can be routed through theirrespective optical paths and ultimately provided as output to the user.

According to some embodiments, the first light beam associated with thefirst image stream can be angularly magnified by optical elements in afirst optical path so that the first image stream can be presented witha wider FOV and lower angular resolution (as governed by the Lagrangeinvariant); whereas the second light beam associated with the secondimage stream is not angularly magnified, demagnified, or magnified by anamount less than the amount of magnification applied to the first lightbeam associated with the first image stream. In this way, the secondimage stream can be presented with a narrower FOV and higher angularresolution (as governed by the Lagrange invariant) than the first imagestream.

FIG. 24 shows a visual field diagram depicting the outer perimeter of anexemplary monocular field of view 3002 for a human eye intwo-dimensional angular space. As shown in FIG. 24 , temporal-nasal andinferior-superior axes of the visual field diagram serve to define thetwo-dimensional angular space within which the outer perimeter of themonocular field of view 3002 is mapped. In this way, the visual fielddiagram of FIG. 24 may be seen as being equivalent or similar to a“Goldmann” visual field map or plot for a human eye. As indicated by thedepicted arrangement of the temporal-nasal and inferior-superior axes,the visual field diagram shown in FIG. 24 represents a visual fielddiagram for the left eye of a human. While field of view can varyslightly from person to person, the depicted field of view is close towhat many humans are capable of viewing with their left eye. It followsthat a visual field diagram depicting the outer perimeter of anexemplary monocular field of view of the right eye might resemblesomething of a version of the visual field diagram of FIG. 24 in whichthe temporal-nasal axis and the outer perimeter of the monocular fieldof view 3002 have been mirrored about the inferior-superior axis.

The visual field diagram of FIG. 24 further depicts the outer perimeterof an exemplary field of regard 3004 for the human eye, which representsa portion of the monocular field of view 30022 in angular space withinwhich the person can fixate. In addition, the visual field diagram ofFIG. 24 also depicts the outer perimeter of an exemplary foveal field3006 for the human eye, which represents a portion of the monocularfield of view 3002 in angular space in direct view of the fovea of thehuman eye at a given point in time. As depicted, a person's foveal field3006 can move anywhere within field of regard 3004. Portions of themonocular field of view 3002 outside of foveal field 3006 in angularspace can be referred herein as the peripheral region of the person'sfield of view. Because of the ability of human eyes to distinguish ahigh level of detail outside of the foveal field 3006 is quite limited,displaying reduced resolution imagery outside of the foveal field 3006is unlikely to be noticed and can allow for substantial savings on powerexpenditure for processing components responsible for generating contentfor the display.

FIG. 25A shows an exemplary wearable display device 4050 configured toprovide virtual content to a user according to some embodiments.Wearable display device 4050 includes main displays 4052 supported byframe 4054. Frame 4054 can be attached to the head of a user using anattachment member taking the form of temple arms 4006.

Referring now to FIG. 25B, an exemplary embodiment of an AR systemconfigured to provide virtual content to a user will now be described.In some embodiments, the AR system of FIG. 25B may represent a system towhich the wearable display device 4050 of FIG. 25A belongs. The ARsystem of FIG. 25B uses stacked light-guiding optical element assemblies4000 and generally includes an image generating processor 4010, a lightsource 4020, a controller 4030, a spatial light modulator (“SLM”) 4040,an injection optical system 4060, and at least one set of stackedeyepiece layers or light guiding optical elements (“LOEs”; e.g., aplanar waveguide) 4000 that functions as a multiple plane focus system.The system may also include an eye-tracking subsystem 4070. It should beappreciated that other embodiments may have multiple sets of stackedLOEs 4000, but the following disclosure will focus on the exemplaryembodiment of FIG. 25B.

The image generating processor 4010 is configured to generate virtualcontent to be displayed to the user. The image generating processor mayconvert an image or video associated with the virtual content to aformat that can be projected to the user in 3-D. For example, ingenerating 3-D content, the virtual content may need to be formattedsuch that portions of a particular image are displayed at a particulardepth plane while others are displayed at other depth planes. In oneembodiment, all of the image may be generated at a particular depthplane. In another embodiment, the image generating processor may beprogrammed to provide slightly different images to the right and lefteyes 210 such that when viewed together, the virtual content appearscoherent and comfortable to the user's eyes.

The image generating processor 4010 may further include a memory 4012, aGPU 4014, a CPU 4016, and other circuitry for image generation andprocessing. The image generating processor 4010 may be programmed withthe desired virtual content to be presented to the user of the AR systemof FIG. 25B. It should be appreciated that in some embodiments, theimage generating processor 4010 may be housed in the wearable AR system.In other embodiments, the image generating processor 4010 and othercircuitry may be housed in a belt pack that is coupled to the wearableoptics. The image generating processor 4010 is operatively coupled tothe light source 4020 which projects the light associated with thedesired virtual content and one or more spatial light modulators(described below).

The light source 4020 is compact and has high resolution. The lightsource 4020 includes a plurality of spatially separated sub-lightsources 4022 that are operatively coupled to a controller 4030(described below). For instance, the light source 4020 may include colorspecific LEDs and lasers disposed in various geometric configurations.Alternatively, the light source 4020 may include LEDs or lasers of likecolor, each one linked to a specific region of the field of view of thedisplay. In another embodiment, the light source 4020 may comprise abroad-area emitter such as an incandescent or fluorescent lamp with amask overlay for segmentation of emission areas and positions. Althoughthe sub-light sources 4022 are directly connected to the AR system ofFIG. 2B in FIG. 2B, the sub-light sources 222 may be connected to systemvia optical fibers (not shown), as long as the distal ends of theoptical fibers (away from the sub-light sources 4022) are spatiallyseparated from each other. The system may also include condenser (notshown) configured to collimate the light from the light source 4020.

The SLM 4040 may be reflective (e.g., a DLP DMD, a MEMS mirror system,an LCOS, or an FLCOS), transmissive (e.g., an LCD) or emissive (e.g. anFSD or an OLED) in various exemplary embodiments. The type of spatiallight modulator (e.g., speed, size, etc.) can be selected to improve thecreation of the 3-D perception. While DLP DMDs operating at higherrefresh rates may be easily incorporated into stationary AR systems,wearable AR systems typically use DLPs of smaller size and power. Thepower of the DLP changes how 3-D depth planes/focal planes are created.The image generating processor 4010 is operatively coupled to the SLM4040, which encodes the light from the light source 4020 with thedesired virtual content. Light from the light source 4020 may be encodedwith the image information when it reflects off of, emits from, orpasses through the SLM 4040.

Referring back to FIG. 25B, the AR system also includes an injectionoptical system 4060 configured to direct the light from the light source4020 (i.e., the plurality of spatially separated sub-light sources 4022)and the SLM 4040 to the LOE assembly 4000. The injection optical system4060 may include one or more lenses that are configured to direct thelight into the LOE assembly 4000. The injection optical system 4060 isconfigured to form spatially separated and distinct pupils (atrespective focal points of the beams exiting from the injection opticalsystem 4060) adjacent the LOEs 4000 corresponding to spatially separatedand distinct beams from the sub-light sources 4022 of the light source4020. The injection optical system 4060 is configured such that thepupils are spatially displaced from each other. In some embodiments, theinjection optical system 4060 is configured to spatially displace thebeams in the X and Y directions only. In such embodiments, the pupilsare formed in one X, Y plane. In other embodiments, the injectionoptical system 4060 is configured to spatially displace the beams in theX, Y and Z directions.

Spatial separation of light beams forms distinct beams and pupils, whichallows placement of in-coupling gratings in distinct beam paths, so thateach in-coupling grating is mostly addressed (e.g., intersected orimpinged) by only one distinct beam (or group of beams). This, in turn,facilitates entry of the spatially separated light beams into respectiveLOEs 4000 of the LOE assembly 4000, while minimizing entry of otherlight beams from other sub-light sources 4022 of the plurality (i.e.,cross-talk). A light beam from a particular sub-light source 4022 entersa respective LOE 4000 through an in-coupling grating (not shown in FIG.25B, see FIGS. 24-26 ) thereon. The in-coupling gratings of respectiveLOEs 4000 are configured to interact with the spatially separated lightbeams from the plurality of sub-light sources 4022 such that eachspatially separated light beam only intersects with the in-couplinggrating of one LOE 4000. Therefore, each spatially separated light beammainly enters one LOE 4000. Accordingly, image data encoded on lightbeams from each of the sub-light sources 4022 by the SLM 4040 can beeffectively propagated along a single LOE 4000 for delivery to an eye210 of a user.

Each LOE 4000 is then configured to project an image or sub-image thatappears to originate from a desired depth plane or FOV angular positiononto a user's retina. The respective pluralities of LOEs 4000 andsub-light sources 4022 can therefore selectively project images(synchronously encoded by the SLM 4040 under the control of controller4030) that appear to originate from various depth planes or positions inspace. By sequentially projecting images using each of the respectivepluralities of LOEs 4000 and sub-light sources 4022 at a sufficientlyhigh frame rate (e.g., 360 Hz for six depth planes at an effectivefull-volume frame rate of 60 Hz), the system of FIG. 25B can generate a3-D image of virtual objects at various depth planes that appear toexist simultaneously in the 3-D image.

The controller 4030 is in communication with and operatively coupled tothe image generating processor 4010, the light source 4020 (sub-lightsources 4022) and the SLM 4040 to coordinate the synchronous display ofimages by instructing the SLM 4040 to encode the light beams from thesub-light sources 4022 with appropriate image information from the imagegenerating processor 4010.

The AR system also includes an optional eye-tracking subsystem 4070 thatis configured to track the user's eyes 4002 and determine the user'sfocus. In one embodiment, only a subset of sub-light sources 4022 may beactivated, based on input from the eye-tracking subsystem, to illuminatea subset of LOEs 4000, as will be discussed below. Based on input fromthe eye-tracking subsystem 4070, one or more sub-light sources 4022corresponding to a particular LOE 4000 may be activated such that theimage is generated at a desired depth plane that coincides with theuser's focus/accommodation. For example, if the users eyes 210 areparallel to each other, the AR system of FIG. 25B may activate thesub-light sources 4022 corresponding to the LOE 4000 that is configuredto deliver collimated light to the user's eyes, such that the imageappears to originate from optical infinity. In another example, if theeye-tracking sub-system 4070 determines that the user's focus is at 1meter away, the sub-light sources 4022 corresponding to the LOE 4000that is configured to focus approximately within that range may beactivated instead. It should be appreciated that, in this particularembodiment, only one group of sub-light sources 4022 is activated at anygiven time, while the other sub-light sources 4020 are deactivated toconserve power.

FIG. 25C illustrates schematically the light paths in an exemplaryviewing optics assembly (VOA) that may be used to present a digital orvirtual image to a viewer, according to some embodiments. In someembodiments, the VOA could be incorporated in a system similar towearable display device 4050 as depicted in FIG. 25A. The VOA includes aprojector 4001 and an eyepiece 200 that may be worn around a viewer'seye. The eyepiece 4000 may, for example, may correspond to LOEs 4000 asdescribed above with reference to FIG. 25B. In some embodiments, theprojector 4001 may include a group of red LEDs, a group of green LEDs,and a group of blue LEDs. For example, the projector 201 may include twored LEDs, two green LEDs, and two blue LEDs according to an embodiment.In some examples, the projector 4001 and components thereof as depictedin FIG. 25C (e.g., LED light source, reflective collimator, LCoS SLM,and projector relay) may represent or provide the functionality of oneor more of light source 4020, sub-light sources 4022, SLM 4040, andinjection optical system 4060, as described above with reference to FIG.25B. The eyepiece 4000 may include one or more eyepiece layers, each ofwhich may represent one of LOEs 4000 as described above with referenceto FIG. 25B. Each eyepiece layer of the eyepiece 4000 may be configuredto project an image or sub-image that appears to originate from arespective desired depth plane or FOV angular position onto the retinaof a viewer's eye.

In one embodiment, the eyepiece 4000 includes three eyepiece layers, oneeyepiece layer for each of the three primary colors, red, green, andblue. For example, in this embodiment, each eyepiece layer of theeyepiece 4000 may be configured to deliver collimated light to the eyethat appears to originate from the optical infinity depth plane (0diopters). In another embodiment, the eyepiece 4000 may include sixeyepiece layers, i.e., one set of eyepiece layers for each of the threeprimary colors configured for forming a virtual image at one depthplane, and another set of eyepiece layers for each of the three primarycolors configured for forming a virtual image at another depth plane.For example, in this embodiment, each eyepiece layer in one set ofeyepiece layers of the eyepiece 4000 may be configured to delivercollimated light to the eye that appears to originate from the opticalinfinity depth plane (0 diopters), while each eyepiece layer in anotherset of eyepiece layers of the eyepiece 4000 may be configured to delivercollimated light to the eye that appears to originate from a distance of2 meters (0.5 diopter). In other embodiments, the eyepiece 4000 mayinclude three or more eyepiece layers for each of the three primarycolors for three or more different depth planes. For instance, in suchembodiments, yet another set of eyepiece layers may each be configuredto deliver collimated light that appears to originate from a distance of1 meter (1 diopter).

Each eyepiece layer comprises a planar waveguide and may include anincoupling grating 4007, an orthogonal pupil expander (OPE) region 4008,and an exit pupil expander (EPE) region 4009. More details aboutincoupling grating, orthogonal pupil expansion, and exit pupil expansionare described in U.S. patent application Ser. No. 14/555,585 and U.S.patent application Ser. No. 14/726,424, the contents of which are herebyexpressly and fully incorporated by reference in their entirety, asthough set forth in full. Still referring to FIG. 25C, the projector4001 projects image light onto the incoupling grating 4007 in aneyepiece layer 4000. The incoupling grating 4007 couples the image lightfrom the projector 4001 into the waveguide propagating in a directiontoward the OPE region 4008. The waveguide propagates the image light inthe horizontal direction by total internal reflection (TIR). The OPEregion 4008 of the eyepiece layer 4000 also includes a diffractiveelement that couples and redirects a portion of the image lightpropagating in the waveguide toward the EPE region 4009. Morespecifically, collimated light propagates horizontally (i.e., relativeto view of FIG. 25C) along the waveguide by TIR, and in doing sorepeatedly intersects with the diffractive element of the OPE region4008. In some examples, the diffractive element of the OPE region 4008has a relatively low diffraction efficiency. This causes a fraction(e.g., 10%) of the light to be diffracted vertically downward toward theEPE region 4009 at each point of intersection with the diffractiveelement of the OPE region 4008, and a fraction of the light to continueon its original trajectory horizontally along the waveguide via TIR. Inthis way, at each point of intersection with the diffractive element ofthe OPE region 4008, additional light is diffracted downward toward theEPE region 4009. By dividing the incoming light into multiple outcoupledsets, the exit pupil of the light is expanded horizontally by thediffractive element of the OPE region 4008. The expanded light coupledout of the OPE region 4008 enters the EPE region 4009.

The EPE region 4009 of the eyepiece layer 4000 also includes adiffractive element that couples and redirects a portion of the imagelight propagating in the waveguide toward a viewer's eye 210. Lightentering the EPE region 4009 propagates vertically (i.e., relative toview of FIG. 25C) along the waveguide by TIR. At each point ofintersection between the propagating light and the diffractive elementof the EPE region 4009, a fraction of the light is diffracted toward theadjacent face of the waveguide allowing the light to escape the TIR,emerge from the face of the waveguide, and propagate toward the viewer'seye 210. In this fashion, an image projected by projector 4001 may beviewed by the viewer's eye 210. In some embodiments, the diffractiveelement of the EPE region 4009 may be designed or configured to have aphase profile that is a summation of a linear diffraction grating and aradially symmetric diffractive lens. The radially symmetric lens aspectof the diffractive element of the EPE region 4009 additionally imparts afocus level to the diffracted light, both shaping the light wavefront(e.g., imparting a curvature) of the individual beam as well as steeringthe beam at an angle that matches the designed focus level. Each beam oflight outcoupled by the diffractive element of the EPE region 4009 mayextend geometrically to a respective focus point positioned in front ofthe viewer, and may be imparted with a convex wavefront profile with acenter of radius at the respective focus point to produce an image orvirtual object at a given focal plane.

Descriptions of such a viewing optics assembly and other similar set-upsare further provided in U.S. patent application Ser. No. 14/331,218,U.S. patent application Ser. No. 15/146,296, and U.S. patent applicationSer. No. 14/555,585, all of which are incorporated by reference hereinin their entireties. It follows that, in some embodiments, the exemplaryVOA may include and/or take on the form of one or more componentsdescribed in any of the patent applications mentioned above withreference to FIG. 25C and incorporated herein by reference.

III. High Field of View and High Resolution Foveated Display UsingMultiple Optical Paths

FIGS. 26A-26D illustrate exemplary render perspectives to be used andlight fields to be produced in an AR system for each of two exemplaryeye orientations. In FIG. 26A, a viewer's eye 210 is oriented in a firstmanner with respect to an eyepiece 5000. In some embodiments, theeyepiece 5000 may be similar to the stack of LOEs or eyepiece 4000 asdescribed above with reference to FIGS. 25B and 25C. More specifically,in this example, the viewer's eye 210 is oriented such that the viewermay be able to see the eyepiece 5000 in a relatively straightforwarddirection. The AR system to which the eyepiece 5000 belongs, which insome examples may be similar to the AR system as described above withreference to FIG. 25B, may perform one or more operations to presentvirtual content on one or more depth planes positioned within theviewer's FOV at one or more distances in front of the viewer's eye 210.

The AR system may determine a perspective within render space from whichthe viewer is to view 3-D virtual contents of the render space, such asvirtual objects, based on the position and orientation of the viewer'shead. As described in further detail below with reference to FIG. 29A,in some embodiments, such an AR system may include one or more sensorsand leverage data from these one or more sensors to determine theposition and/or orientation of the viewer's head. The AR system mayinclude such one or more sensors in addition to one or more eye-trackingcomponents, such as one or more components of the eye-trackingsub-system 4070 described above with reference to FIG. 25B. With suchdata, the AR system may effectively map the position and orientation ofthe viewer's head within the real world to a particular location and aparticular angular position within a 3D virtual environment, create avirtual camera that is positioned at the particular location within the3D virtual environment and oriented at the particular angular positionwithin the 3D virtual environment relative to at the particular locationwithin the 3D virtual environment, and render virtual content for theviewer as it would be captured by the virtual camera. Further detailsdiscussing real world to virtual world mapping processes are provided inU.S. patent application Ser. No. 15/296,869, entitled “SELECTING VIRTUALOBJECTS IN A THREE-DIMENSIONAL SPACE,” which is expressly incorporatedherein by reference in its entirety for all purposes.

In some examples, the AR system may create or dynamically repositionand/or reorient one such head-tracked virtual camera for the viewer'sleft eye or eye socket, and another such head-tracked virtual camera forthe viewer's right eye or eye socket, as the viewer's eyes and or eyesockets are physically separated from one another and thus consistentlypositioned at different locations. It follows that virtual contentrendered from the perspective of a head-tracked virtual cameraassociated with the viewer's left eye or eye socket may be presented tothe viewer through an eyepiece on the left side of a wearable displaydevice, such as that described above with reference to FIGS. 25A-25C,and that virtual content rendered from the perspective of a head-trackedvirtual camera associated with the viewer's right eye or eye socket maybe presented to the viewer through an eyepiece on the right side of thewearable display device. Although a head-tracked virtual camera may becreated and/or dynamically repositioned for each eye or eye socket basedon information regarding the current position and orientation of theviewer's head, the position and orientation of such a head-trackedvirtual camera may neither depend upon the position nor the orientationof each eye of the viewer relative to the respective eye socket of theviewer or the viewer's head. Further details discussing the creation,adjustment, and use of virtual cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

The AR system of FIG. 26A may create or dynamically reposition and/orreorient such a head-tracked virtual camera, render virtual content fromthe perspective of the head-tracked virtual camera (perspective 5010),and project light representing renderings of the virtual content throughthe eyepiece 5000 and onto the retina of the viewer's eye 210. As shownin FIG. 26A, the head-tracked render perspective 5010 may provide an FOVspanning a region of ±θ₃₁₀ angular units diagonally, horizontally,and/or vertically. As described in further detail below, in someembodiments, the head-tracked render perspective 5010 may provide arelatively wide FOV. In such embodiments, the AR system may also createor dynamically reposition and/or reorient another virtual camera foreach eye or eye socket different from and in addition to a head-trackedvirtual camera. In the example of FIG. 26A, the AR system may render andpresent virtual content from the perspective of the head-tracked virtualcamera 5010 along with virtual content from the perspective of anothervirtual camera in render space.

For instance, in such embodiments, the AR system of FIG. 26A may createor dynamically reposition and/or reorient such a fovea-tracked virtualcamera based on the current gaze of the viewer's eye 210. As describedin further detail below with reference to FIG. 29A, in some examples,such an AR system may include one or more eye-tracking components, suchas one or more components of the eye-tracking sub-system 4070 describedabove with reference to FIG. 25B, to determine the viewer's currentgaze, the current position and/or orientation of the viewer's eye 210relative to the viewer's head, and the like. With such data, the ARsystem of FIG. 26A may create or dynamically reposition and/or reorientsuch a fovea-tracked virtual camera, render virtual content from theperspective of the fovea-tracked virtual camera (perspective 5020A), andproject light representing virtual content as rendered from perspective5020A through the eyepiece 5000 and onto the fovea of the viewer's eye210.

As shown in FIG. 26A, the fovea-tracked render perspective 5020A mayprovide for an FOV that is narrower than that of the head-tracked renderperspective 5010. In this way, the FOV of the fovea-tracked renderperspective 5020A can be seen as occupying a conical subspace of the FOVof the head-tracked render perspective 5010. That is, the FOV of thefovea-tracked render perspective 5020A may be a subfield of the FOV ofthe head-tracked render perspective 5010. For instance, as shown in FIG.26A, the fovea-tracked render perspective 320A may provide an FOVspanning a region of ±θ_(320A) angular units diagonally, horizontally,and/or vertically, such that the relationship between the FOV of thehead-tracked render perspective 5010 and the fovea-tracked renderperspective 5020A is given by −θ₃₁₀≤−θ_(320A)≤θ_(320A)≤θ₃₁₀. In someexamples, the FOV of the head-tracked render perspective 5010 may be atleast as wide as the viewer's field of regard, which in this examplewould be the total conical space within which the viewer's eye 210 canfixate when the viewer's head is held in a given position andorientation. As such, in these examples, the head-tracked virtual cameraand the fovea-tracked virtual camera may be positioned at substantiallythe same location within render space or may be positioned at locationswithin render space that are a fixed distance away from one another,such that both virtual cameras may be linearly and/or angularlytranslated in unison within render space when the position and/ororientation of the viewer's head changes. For example, the head-trackedvirtual camera may be positioned at a location in render space thatcorresponds to the center-of-rotation of the viewer's eye 210, while thefovea-tracked virtual camera may be positioned at a location in renderspace that corresponds to a region of the viewer's eye 210 between thecenter-of-rotation and cornea. Indeed, the Euclidean distance betweenthe two virtual cameras may remain substantially constant whentranslated in render space in much the same way that the Euclideandistance between two specific regions of the viewer's eye 210 or anotherrigid body may remain substantially constant at all times.

Although the spatial relationship between each virtual camera in such apair of virtual cameras may remain substantially fixed within renderspace throughout use of the AR system in these examples, the orientationof the fovea-tracked virtual camera may, however, vary relative to thehead-tracked virtual camera when the viewer rotates their eye 210. Inthis way, the conical subspace of the FOV of the head-tracked virtualcamera that is occupied by the FOV of the fovea-tracked virtual cameramay dynamically change as the viewer rotates their eye 210.

Furthermore, virtual objects and other content that fall within thefovea-tracked render perspective 5020A may be rendered and presented bythe AR system in relatively high resolution. More specifically, theresolution at which virtual content within the FOV of the fovea-trackedvirtual camera is rendered and presented may be higher than theresolution at which virtual content within the FOV of the head-trackedvirtual camera is rendered and presented. In this way, thehighest-resolution subfield of a given light field that is outcoupled bythe eyepiece 5000 and projected onto the retina of the viewer's eye 210may be that which reaches the fovea of the viewer's eye 210.

FIG. 3B illustrates an exemplary light field 5030A that is outcoupled bythe eyepiece 5000 and projected onto the retina of the viewer's eye 210while the viewer's eye 210 is oriented in the first manner as depictedin FIG. 26A and described above with reference thereto. The light field5030A may include various angular light components representative ofvirtual content as would be captured in render space by theabovementioned pair of virtual cameras. As described in further detailbelow with reference to FIG. 26A and onward, light representative ofvirtual content as would be captured in render space by the head-trackedvirtual camera and light representative of virtual content as would becaptured in render space by the fovea-tracked virtual camera may bemultiplexed by the AR system according to any of a variety of differentmultiplexing schemes. Employment of such multiplexing schemes may, atleast in some instances, allow for the AR system to operate with greaterefficiency and/or occupy less physical space.

Still referring to FIG. 26B, angular light components of the light field5030A that are representative of virtual content as would be captured inrender space by the head-tracked virtual camera (e.g., virtual objectsand other content that fall within the head-tracked render perspective5010) may include those which are to be projected onto the retina of theviewer's eye 210 at angles ranging from −θ₃₁₀ to +θ₃₁₀ angular unitsrelative to the viewer's eye 210. Similarly, angular light components ofthe light field 5030A that are representative of virtual content aswould be captured in render space by the fovea-tracked virtual camera(e.g., virtual objects and other content that fall within thefovea-tracked render perspective 5020A) may include those which are tobe projected onto the retina of the viewer's eye 210 at angles rangingfrom −θ_(320A) to +θ_(320A) angular units relative to the viewer's eye210. The intervals between −θ_(320A) and +θ_(320A) angular units atwhich such angular light components associated with the fovea-trackedrender perspective 5020A occur within the light field 5030A may behigher in regularity than the intervals between −θ₃₁₀ and +θ₃₁₀ angularunits at which angular light components associated with the head-trackedrender perspective 5010 occur within the light field 5030A. In this way,the resolution at which virtual content associated with thefovea-tracked render perspective 5020A may be rendered and presented tothe viewer may be higher than the resolution at which virtual contentassociated with the head-tracked render perspective 5010 may be renderedand presented to the viewer.

In some embodiments, angular light components associated with thehead-tracked render perspective 5010 that occur within the light field5030A may further include those which are to be projected onto theretina of the viewer's eye 210 at angles ranging from −θ_(320A) to+θ_(320A) angular units relative to the viewer's eye 210. In suchembodiments, the intervals between −θ_(320A) and +θ_(320A) angular unitsat which such angular light components associated with the head-trackedrender perspective 5010 occur within the light field 5030A may be lowerin regularity than the intervals between −θ_(320A) and +θ_(320A) angularunits at which angular light components associated with thefovea-tracked render perspective 5020A occur within the light field5030A. In other embodiments, angular light components associated withthe head-tracked render perspective 5010 that occur within the lightfield 5030A may exclude those which are to be projected onto the retinaof the viewer's eye 210 at angles ranging from −θ_(320A) to +θ_(320A)angular units relative to the viewer's eye 210. As such, in these otherembodiments, angular light components associated with the head-trackedrender perspective 5010 that occur within the light field 5030A may bethose which are to be projected onto the retina of the viewer's eye 210at angles between −θ₃₁₀ and −θ_(320A) angular units or angles betweenθ_(320A) and θ₃₁₀.

In FIG. 26C, the viewer's eye 210 is oriented in a second manner withrespect to the eyepiece 5000 different from the first manner in whichthe viewer's eye 210 is oriented with respect to the eyepiece 5000 inFIGS. 26A-26B. For purposes of example, the position and orientation ofthe viewer's head in FIGS. 26C-26D may be treated as being the same asthe position and orientation of the viewer's head as described abovewith reference to FIGS. 26A-26B. As such, FIGS. 26A-26B and FIGS.26C-26D may represent the abovementioned viewer and AR system in firstand second time-sequential stages, respectively. More specifically, inthis example, the viewer's eye 210 has rotated off-center from therelatively straightforward orientation as depicted in FIGS. 26A-26B.

In transitioning from the first stage to the second stage, the AR systemof FIG. 26C may, for instance, function to maintain the head-trackedvirtual camera at the same position and orientation as described abovewith reference to FIGS. 26A-26B, as the viewer's head pose (e.g.,position and orientation) has not changed. As such, in the second stagedepicted in FIGS. 26C-26D, the AR system may render virtual content fromthe perspective of the head-tracked virtual camera (i.e., head-trackedrender perspective 5010) and project light representing renderings ofthe virtual content through the eyepiece 5000 and onto the retina of theviewer's eye 210. While the head-tracked render perspective 5010 mayremain static or relatively static throughout the first and secondtime-sequential stages of FIGS. 26A-26D, in transitioning from the firststage to the second stage, the AR system may function to adjust theorientation of a fovea-tracked virtual camera in render space based onthe change in gaze of the viewer's eye 210 from the first stage to thesecond stage. That is, the AR system may replace or reorient thefovea-tracked virtual camera as employed in the first stage to providethe fovea-tracked render perspective 5020A, such that the fovea-trackedvirtual camera as employed in the second stage provides a fovea-trackedrender perspective 5020C different from the fovea-tracked renderperspective 5020A. It follows that, in the second stage, the AR systemmay also render virtual content from the perspective of thefovea-tracked virtual camera perspective 5020C and project lightrepresenting renderings of the virtual content through the eyepiece 5000and onto the fovea of the viewer's eye 201.

In the example of FIGS. 26C-26D, the fovea-tracked render perspective5020C may occupy a different conical subspace of the head-tracked renderperspective 5010 than that of the fovea-tracked render perspective5020A. For instance, as shown in FIG. 26C, the fovea-tracked renderperspective 5020C may provide an FOV displaced θ_(320C) angular unitsfrom the FOV of the fovea-tracked render perspective 5020A and spanninga region of ±θ_(320A) angular units diagonally, horizontally, and/orvertically. That is, the fovea-tracked render perspective 5020C mayprovide an FOV spanning a region of θ_(320C)±θ_(320A) angular unitsdiagonally, horizontally, and/or vertically.

FIG. 26D illustrates an exemplary light field 5030C that is outcoupledby the eyepiece 5000 and projected onto the retina of the viewer's eye201 while the viewer's eye 201 is oriented in the second manner asdepicted in FIG. 26C and described above with reference thereto. Thelight field 5030C may include various angular light componentsrepresentative of virtual content as would be captured in render spacefrom the head-tracked render perspective 5010 and the fovea-trackedrender perspective 5020C. Angular light components of the light field5030C that are representative of virtual content as would be captured inrender space from the head-tracked render perspective 5010 may includethose which are to be projected onto the retina of the viewer's eye 210at angles ranging from −θ₃₁₀ to +θ₃₁₀ angular units relative to theviewer's eye 210. However, in a departure from the first stage asdescribed above with reference to FIGS. 26A-26B, the angular lightcomponents of light field 5030C that are representative of virtualcontent as would be captured in render space by the fovea-trackedvirtual camera (e.g., virtual objects and other content that fall withinthe fovea-tracked render perspective 5020C) may include those which areto be projected onto the retina of the viewer's eye 210 at anglesranging from θ_(320C)-θ_(320A) angular units to θ_(320C)+θ_(320A)angular units relative to the viewer's eye 210.

The intervals between θ_(320C)-θ_(320A) angular units andθ_(320C)+θ_(320A) angular units at which such angular light componentsassociated with the fovea-tracked render perspective 320C occur withinthe light field 5030C may be higher than the intervals between −θ₃₁₀ and+θ₃₁₀ angular units at which angular light components associated withthe head-tracked render perspective 5010 occur within the light field5030C. In this way, the resolution at which virtual content associatedwith the fovea-tracked render perspective 5020C may be rendered andpresented to the viewer may be higher than the resolution at whichvirtual content associated with the head-tracked render perspective 5010may be rendered and presented to the viewer, which notably includesvirtual content represented by angular light components that are to beprojected onto the retina of the viewer's eye 210 at angles ranging from−θ_(320A) to +θ_(320A) angular units relative to the viewer's eye 210.

In some embodiments, angular light components associated with thehead-tracked render perspective 5010 that occur within the light field5030C may further include those which are to be projected onto theretina of the viewer's eye 210 at angles ranging from θ_(320C)−θ_(320A)angular units and θ_(320C)+θ_(320A) angular units relative to theviewer's eye 210. In such embodiments, the intervals between−θ_(320C)−θ_(320A) angular units and θ_(320C)+θ_(320A) angular units atwhich such angular light components associated with the head-trackedrender perspective 310 occur within the light field 5030C may be lowerin regularity than the intervals between θ_(320C)−θ_(320A) angular unitsand θ_(320C)+θ_(320A) angular units angular units at which angular lightcomponents associated with the fovea-tracked render perspective 5020Coccur within the light field 5030C. In other embodiments, angular lightcomponents associated with the head-tracked render perspective 5010 thatoccur within the light field 5030C may exclude those which are to beprojected onto the retina of the viewer's eye 210 at angles ranging fromθ_(320C)−θ_(320A) angular units and θ_(320C)+θ_(320A) angular unitsrelative to the viewer's eye 210. As such, in these other embodiments,angular light components associated with the head-tracked renderperspective 5010 that occur within the light field 5030C may be thosewhich are to be projected onto the retina of the viewer's eye 210 atangles between −θ₃₁₀ and θ_(320C)−θ_(320A) angular units and angularunits or angles between θ_(320C)+θ_(320A) angular and θ₃₁₀ angularunits.

FIGS. 26E-26F illustrate schematically an exemplary configuration ofimages that can be presented to a user according to some embodiments. Itshould be noted that the grid squares in FIGS. 26E-26F representschematically image points that, much like fields 3002, 3004 and 3006 asdescribed above with reference to FIG. 24 , are defined intwo-dimensional angular space. A low-resolution first image stream 5010Ehaving a wide FOV can be displayed at a static location. Alow-resolution first image stream 5010E having a wide FOV can representone or more images of virtual content as would be captured by a firstvirtual camera having a static position and orientation in render space.For instance, the low-resolution first image stream 5010E can representone or more images of virtual content as would be captured by ahead-tracked virtual camera such as the head-tracked virtual cameradescribed above with reference to FIGS. 26A-26D. The first image stream5010E can encompass the user's vision to evoke an immersion experienceto the user.

A high-resolution second image stream 5020E having a relatively narrowFOV can be displayed within the boundaries of the first image stream5010E. In some examples, the second image stream 5020E can represent oneor more images of virtual content as would be captured by a second,different virtual camera having an orientation in render space that canbe dynamically adjusted in real-time based on data obtained usingeye-gaze tracking techniques to angular positions coinciding with theuser's current fixation point. In these examples, the high-resolutionsecond image stream 5020E can represent one or more images of virtualcontent as would be captured by a fovea-tracked virtual camera such asthe fovea-tracked virtual camera described above with reference to FIGS.26A-26D. In other words, the perspective in render space from which oneor more images of virtual content represented by the second image stream5020E is captured can be reoriented as the user's eye gaze changes, suchthat the perspective associated with the second image stream 5020E ispersistently aligned with the user's foveal vision.

For example, the second image stream 5020E can encompass virtual contentlocated within a first region of render space when the user's eye gazeis fixed at the first position as illustrated in FIG. 26E. As the user'seye gaze moves to a second position different from the first position,the perspective associated with the second image stream 5020E can beadjusted such that the second image stream 5020E can encompass virtualcontent located within a second region of render space, as illustratedin FIG. 26F. In some embodiments, the first image stream 5010E has awide FOV, but a low angular resolution as indicated by the coarse grid.The second image stream 5020E has a narrow FOV, but a high angularresolution as indicated by the fine grid.

FIG. 26G illustrates schematically an exemplary configuration of imagesthat can be presented to a user according to some other embodiments.Like FIGS. 26E-26F, the grid squares in FIG. 26G represent schematicallyimage points that are defined in two-dimensional angular space. Similarto the configuration illustrated in FIGS. 26E-26F, a low resolutionfirst image stream 5010G having a wide FOV encompasses virtual contentas viewed from a head-tracked render perspective, while a highresolution second image stream 5020G having a narrow FOV encompassesvirtual content as viewed from a fovea-tracked render perspective thatmay be dynamically reoriented so as to coincide with the user's currentfixation point. Here, the outer perimeter of the FOV associated with thefirst image stream 5010G can form a rectangular boundary with roundedcorners, and the outer perimeter of the FOV associated with the secondimage stream 5020G can form a circular boundary.

FIG. 26H illustrates schematically an exemplary configuration of imagesthat can be presented to a user according to yet some other embodiments.Like FIGS. 26E-26G, the grid squares in FIG. 26H represent schematicallyimage points that are defined in two-dimensional angular space. Here,both the outer perimeter of the FOV associated with the first imagestream 5010H and the outer perimeter of the FOV associated with thesecond image stream 5020H can form circular boundaries. In some otherembodiments, either the outer perimeter of the FOV associated with thefirst image stream 5010H and the outer perimeter of the FOV associatedwith the second image stream 5020H, or both, can form an ellipticalboundary or other shapes. In some embodiments, an image source of the ARsystem of FIG. 26H may include a scanning fiber that can be scanned in apredetermined pattern to provide light beams for the first image stream5010H and the second image stream 5020H with desired boundary shapes.

FIG. 27 illustrates a field of view 3002 and a field of regard 3004 asshown in FIG. 24 , overlaid upon one of the displays 4052 in thewearable display device 4050 as shown in FIG. 25A. According to someembodiments, the wide FOV and low resolution first image stream 5010Eillustrated in FIGS. 26E-26F can be displayed across the entire area ofthe display 4052 (the relatively low resolution of the first imagestream 5010E is illustrated with a coarse grid), while the narrow FOVand high resolution second image stream 5020E can be displayed at theuser's current foveated region 3006 (the relatively high resolution ofthe second image stream 5020E is illustrated with a fine grid). While inFIG. 27 the first image stream 5010E and the second image stream 5020Eare illustrated as displayed in the “plane” of the displays 4052, in asee-through augmented reality (AR) display system the first image stream5010E and the second image stream 5020E can also be presented to theuser as light fields within certain angular fields of view. Such an ARdisplay system can produce display planes that appear to be “floating”at some distance (e.g., 2 meters) in front of the user. The displayplane can appear to be much larger than the glasses. This floatingdistanced display is used for overlaying information on the real world.

FIGS. 28A-28B illustrate some of the principles described in FIGS.26A-26D using exemplary virtual content that can be presented to a useraccording to some embodiments. As such, FIGS. 28A-28B may represent aviewer and an AR system in first and second time-sequential stages,respectively. Furthermore, some or all of the components shown in FIGS.28A-28B may be the same as or at least similar to components asdescribed above with reference to FIGS. 26A-26D.

The AR system of FIGS. 28A-28B may create or dynamically repositionand/or reorient a head-tracked virtual camera similar to thehead-tracked virtual camera described above with reference to FIGS.26A-26D, render virtual content from the perspective of the head-trackedvirtual camera, and project light representing renderings of the virtualcontent through the eyepiece 6000 and onto the retina of the viewer'seye 210. The AR system of FIGS. 28A-28B may also create or dynamicallyreposition and/or reorient a fovea-tracked virtual camera similar to thefovea-tracked virtual camera described above with reference to FIGS.26A-26D, render virtual content from the perspective of thefovea-tracked virtual camera, and project light representing renderingsof the virtual content through the eyepiece 400 and onto the fovea ofthe viewer's eye 210. As shown in FIGS. 28A-28B, such virtual contentmay include 3-D virtual objects 6011, 6012, and 6013. In some examples,the AR system of FIGS. 28A-28B may perform one or more of the operationsdescribed immediately above regarding the head-tracked renderperspective and one or more of the operations described immediatelyabove regarding the fovea-tracked render perspective simultaneously. Inother examples, the AR system of FIGS. 28A-28B may perform suchoperations in rapid succession.

In this example, the FOV of the head-tracked render perspective employedby the AR system in FIGS. 28A-28B may be diagonally, horizontally,and/or vertically wide enough in angular space to encompass each ofvirtual objects 6011, 6012, and 6013. For purposes of example, theposition and orientation of the viewer's head may be treated as beingstatic throughout the first and second stages as depicted in FIGS. 28Aand 28B, respectively, such that the position and orientation of thehead-tracked render perspective remain the same throughout the twostages. In order for the FOV of the head-tracked render perspectiveemployed by the AR system to be large enough to encompass virtualobjects 6011-6013, it must at least span a region of α+ζ angular unitsdiagonally, horizontally, and/or vertically. More specifically, in theexample of FIGS. 28A-28B, it can be seen that virtual objects 6011,6012, and 6013 may span regions of α−β, γ+δ, and ζ−ε angular units,respectively.

In FIG. 28A, a viewer's eye 210 is oriented in a first manner withrespect to an eyepiece 6000, such that the viewer may be able to see theeyepiece 6000 in a relatively straightforward direction. The orientationof the viewer's eye 210 in FIG. 28A may, for instance, be the same as orsimilar to the orientation of the viewer's eye 210 as described abovewith reference to FIGS. 26A-26B, and may be determined by the AR systemusing one or more of the sensing components and/or techniques describedherein. As such, in the stage depicted in FIG. 28A, the AR system mayemploy head-tracked and fovea-tracked render perspectives at relativepositions and orientations similar to those of the head-tracked andfovea-tracked render perspectives 5010 and 5020A, respectively. In theparticular example of FIG. 28A, the FOV of the fovea-tracked renderperspective employed by the AR system may, for instance, encompassvirtual object 6012, but may not encompass either of virtual objects6011 and 6013. It follows that, in FIG. 28A, the AR system may rendervirtual object 6012 as it would be captured from the perspective of thefovea-tracked virtual camera in high definition, and may render virtualobjects 6011 and 6013 as they would be captured from the perspective ofthe head-tracked virtual camera in lower definition. In addition, the ARsystem may project light representing such renderings of virtual objects6011, 6012, and 6013 through the eyepiece 6000 and onto the retina ofthe viewer's eye 210. In some embodiments, the AR system may also rendervirtual object 6012 as it would be captured from the perspective of thehead-tracked virtual camera in lower definition.

FIG. 28A also illustrates an exemplary light field 6030A that isoutcoupled by the eyepiece 6000 and projected onto the retina of theviewer's eye 210. The light field 6030A may include various angularlight components representative of one or more of the abovementionedrenderings of virtual objects 6011, 6012, and 6013. For example, angularlight components of the light field 6030A that are representative of thevirtual object 6011 as it would be captured from the perspective of thehead-tracked virtual camera may include those which are to be projectedonto the retina of the viewer's eye 210 at angles ranging from −α to −βangular units relative to the viewer's eye 210, and angular lightcomponents of the light field 6030A that are representative of thevirtual object 6013 as it would be captured from the perspective of thehead-tracked virtual camera may include those which are to be projectedonto the retina of the viewer's eye 210 at angles ranging from ε to ζangular units relative to the viewer's eye 210. Similarly, angular lightcomponents of the light field 6030A that are representative of thevirtual object 6012 as it would be captured from the perspective of thefovea-tracked virtual camera may include those which are to be projectedonto the fovea of the viewer's eye 210 at angles ranging from −γ to δangular units relative to the viewer's eye 210. As such, components ofthe light field 6030A that are representative of virtual object 6012(i.e., components to be projected at angles ranging from −γ to δ angularunits relative to the viewer's eye 210) may be more densely distributedin angular space than components of the light field 6030A that arerepresentative of virtual object 6011 or 6013 (i.e., components to beprojected at angles ranging from −α to −β or ε to ζ angular unitsrelative to the viewer's eye 210). In this way, the resolution at whichthe virtual object 6012 may be rendered and presented to the viewer maybe higher than the resolution at which virtual object 6011 or 6013 maybe rendered and presented to the viewer.

In FIG. 28B, the viewer's eye 210 is oriented in a second manner withrespect to the eyepiece 6000 different from the first manner in whichthe viewer's eye 210 is oriented with respect to the eyepiece 6000 inFIG. 28A. The orientation of the viewer's eye 210 in FIG. 28B may, forinstance, be the same as or similar to the orientation of the viewer'seye 210 as described above with reference to FIGS. 26C-26D, and may bedetermined by the AR system using one or more of the sensing componentsand/or techniques described herein. As such, in the stage depicted inFIG. 28B, the AR system may employ head-tracked and fovea-tracked renderperspectives at relative positions and orientations similar to those ofthe head-tracked and fovea-tracked render perspectives 5010 and 5020C,respectively. In the particular example of FIG. 28B, the FOV of thefovea-tracked render perspective employed by the AR system may, forinstance, encompass virtual object 6013, but may not encompass either ofvirtual objects 6011 and 6012. It follows that, in FIG. 28B, the ARsystem may render virtual object 6013 as it would be captured from theperspective of the fovea-tracked virtual camera in high definition, andmay render virtual objects 6011 and 6012 as they would be captured fromthe perspective of the head-tracked virtual camera in lower definition.In addition, the AR system may project light representing suchrenderings of virtual objects 6011, 6012, and 6013 through the eyepiece6000 and onto the retina of the viewer's eye 210. In some embodiments,the AR system may also render virtual object 6013 as it would becaptured from the perspective of the head-tracked virtual camera inlower definition.

FIG. 28B also illustrates an exemplary light field 6030B that isoutcoupled by the eyepiece 6000 and projected onto the retina of theviewer's eye 210. The light field 6030B may include various angularlight components representative of one or more of the abovementionedrenderings of virtual objects 6011, 6012, and 6013. For example, angularlight components of the light field 6030B that are representative of thevirtual object 6011 as it would be captured from the perspective of thehead-tracked virtual camera may include those which are to be projectedonto the retina of the viewer's eye 210 at angles ranging from −α to −βangular units relative to the viewer's eye 210, and angular lightcomponents of the light field 6030B that are representative of thevirtual object 6012 as it would be captured from the perspective of thehead-tracked virtual camera may include those which are to be projectedonto the retina of the viewer's eye 210 at angles ranging from −γ to δangular units relative to the viewer's eye 210. Similarly, angular lightcomponents of the light field 6030B that are representative of thevirtual object 6013 as it would be captured from the perspective of thefovea-tracked virtual camera may include those which are to be projectedonto the fovea of the viewer's eye 210 at angles ranging from ε to ζangular units relative to the viewer's eye 210. As such, components ofthe light field 6030B that are representative of virtual object 6013(i.e., components to be projected at angles ranging from ε to ζ angularunits relative to the viewer's eye 210) may be more densely distributedin angular space than components of the light field 6030A that arerepresentative of virtual object 6011 or 6012 (i.e., components to beprojected at angles ranging from −α to −β or −γ to δ angular unitsrelative to the viewer's eye 210). In this way, the resolution at whichthe virtual object 6013 may be rendered and presented to the viewer maybe higher than the resolution at which virtual object 6011 or 6012 maybe rendered and presented to the viewer. Indeed, from the stage of FIG.28A to the stage of FIG. 28B, the AR system described herein withreference thereto has effectively reoriented the perspective from whichvirtual content may be viewed in high resolution in accordance with thechange in gaze of the viewer's eye 402 between stages.

FIGS. 28C-28F illustrate some of the principles described in FIGS. 3E-3Fusing some exemplary images that can be presented to a user according tosome embodiments. In some examples, the one or more of the images and/orimage streams depicted in FIGS. 28C-28F may represent two-dimensionalimages or portions thereof that are to be displayed at a particulardepth plane, such as one or more of the depth planes described abovewith reference to FIG. 25B. That is, such images and/or image streamsmay represent 3-D virtual content having been projected onto at leastone two-dimensional surface at a fixed distance away from the user. Insuch examples, it is to be understood that such images and/or imagestreams may be presented to the user as one or more light fields withcertain angular fields of view similar to those described above withreference to FIGS. 26A-26D and 28A-28B.

As depicted, a first image stream 6010 includes a tree. During a firstperiod of time represented by FIG. 28C, eye-tracking sensors candetermine a user's eye gaze (i.e., the foveal vision) is focused withina first region 6010-1 of the tree that includes the trunk of the tree.In response to determining the user's eye gaze is focused within thefirst region 6010-1, a second image stream 6020 that includeshigh-resolution imagery associated with the first region 6010-1 of thefirst image stream 6010 can be positioned within the first region 410-1concurrent with the display of the first image stream 6010. The firstimage stream 410 can have a lower resolution than the second imagestream 6020, as illustrated in FIG. 28C.

During a second period of time represented by FIG. 28D, eye-trackingsensors can determine the user's eye gaze has moved to a second region6010-2 of the tree that includes a branch of the tree as illustrated inFIG. 28D. Accordingly, the second image stream 420 can be shifted to thesecond region 6010-2 and have its content changed to correspond to thecontent within second region 6010-2 of the first image stream 6010.Because the higher resolution second image stream 6020 overlays theportion of the first image stream 6010 within the user's foveal vision,the lower resolution of the portion of the first image stream 6010surrounding the second image stream 6020 may not be perceived or noticedby the user. In this way, the user may perceive the combination of thefirst image stream 6010 and the second image stream 6020 as having botha wide FOV and high resolution. Such a display system can afford severaladvantages. For example, the display system can provide a superior userexperience while maintaining a relatively small form factor and keepingcomputation resource requirement relatively low. The small form factorand low computation resource requirement can be due to the device onlyhaving to generate high-resolution imagery in a limited region of thedisplay.

The second image stream 6020 can be overlaid on the first image stream6010 simultaneously, or in rapid succession. As discussed above, in someembodiments, the subset of the content of the first image stream 6010overlaid by the second image stream 6020 can be turned off or bepresented with a lower intensity for more uniform brightness and forbetter resolution perception. It should also be noted that in someembodiments the second image stream associated with the second imagestream 6020 can differ from the first image stream associated with thefirst image stream 6010 in other ways. For example, a color resolutionof the second image stream could be higher than the color resolution ofthe first image stream. A refresh rate of the second image stream couldalso be higher than the refresh rate of the first image stream.

FIG. 28E illustrates an exemplary high-FOV low-resolution image frame(i.e., the first image stream), and FIG. 28F illustrates an exemplarylow-FOV high-resolution image frame (i.e., the second image stream),according to some embodiments. As illustrated in FIG. 28E, the region6030 of the high-FOV low-resolution image frame, which would be overlaidby the low-FOV high-resolution image frame, can be devoid of virtualcontent. By omitting the portion of the high-FOV image that correspondsto region 6030, any image blurring or smearing resulting from slightdifferences in the two images can be avoided. The content of the low-FOVhigh-resolution image frame (e.g., as illustrated in FIG. 28F) caninclude a high resolution version of the content corresponding to region6030.

FIG. 29A shows a simplified block diagram of a display system 7000Aaccording to some embodiments. The display system 7000A can include oneor more sensors 7002 for detecting the position and movement of the headof a user, as well as the eye position and inter-ocular distance of theuser. Such sensors may include image capture devices (such as cameras),microphones, inertial measurement units, accelerometers, compasses, GPSunits, radio devices, gyroscopes, and the like. In an augmented realitysystem, the one or more sensors 7002 can be mounted on a head-wornframe.

For example, in some implementations, the one or more sensors 7002 ofthe display system 7000A may be part of a head worn transducer systemand include one or more inertial transducers to capture inertialmeasures indicative of movement of the head of the user. As such, inthese implementations the one or more sensors 7002 may be used to sense,measure, or collect information about the head movements of the user.For instance, such may be used to detect measurement movements, speeds,acceleration, and/or positions of the head of the user.

In some embodiments, the one or more sensors 7002 can include one ormore forward facing cameras, which may be used to capture informationabout the environment in which the user is located. The forward facingcameras may be used to capture information indicative of distance andorientation of the user with respect to that environment and specificobjects in that environment. When head worn, the forward facing camerasis particularly suited to capture information indicative of distance andorientation of the head of the user with respect to the environment inwhich the user is located and specific objects in that environment. Theforward facing cameras can be employed to detect head movement, speed,and acceleration of head movements. The forward facing cameras can alsobe employed to detect or infer a center of attention of the user, forexample, based at least in part on an orientation of the head of theuser. Orientation may be detected in any direction (e.g., up and down,left and right with respect to the reference frame of the user).

The one or more sensors 7002 can also include a pair of rearward facingcameras to track movement, blinking, and depth of focus of the eyes ofthe user. Such eye-tracking information can, for example, be discernedby projecting light at the user's eyes, and detecting the return orreflection of at least some of that projected light. Further detailsdiscussing eye-tracking devices are provided in U.S. Provisional PatentApplication No. 61/801,219, entitled “DISPLAY SYSTEM AND METHOD,” U.S.Provisional Patent Application No. 62/005,834, entitled “METHODS ANDSYSTEM FOR CREATING FOCAL PLANES IN VIRTUAL AND AUGMENTED REALITY,” U.S.Provisional Patent Application No. 61/776,771, entitled “SYSTEM ANDMETHOD FOR AUGMENTED AND VIRTUAL REALITY,” and U.S. Provisional PatentApplication No. 62/420,292, entitled “METHOD AND SYSTEM FOR EYE TRACKINGUSING SPECKLE PATTERNS,” which are expressly incorporated herein byreference.

The display system 7000A can further include a user orientationdetermination module 7004 communicatively coupled to the one or moresensors 7002. The user orientation determination module 7004 receivesdata from the one or more sensors 7002 and uses such data to determinethe user's head pose, cornea positions, inter-pupillary distance, andthe like. The user orientation determination module 7004 detects theinstantaneous position of the head of the user and may predict theposition of the head of the user based on position data received fromthe one or more sensors 7002. The user orientation determination module7004 also tracks the eyes of the user based on the tracking datareceived from the one or more sensors 7002.

The display system 7000A may further include a control subsystem thatmay take any of a large variety of forms. The control subsystem includesa number of controllers, for instance one or more microcontrollers,microprocessors or central processing units (CPUs), digital signalprocessors, graphics processing units (GPUs), other integrated circuitcontrollers, such as application specific integrated circuits (ASICs),programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/orprogrammable logic controllers (PLUs).

In the example depicted in FIG. 29A, the display system 7000A includes acentral processing unit (CPU) 7010, a graphics processing unit (GPU)7020, and frame buffers 7042 and 7044. Briefly, and as described infurther detail below, the CPU 7010 controls overall operation, while theGPU 7020 renders frames (i.e., translating a three-dimensional sceneinto a two-dimensional image) from three-dimensional data stored indatabase 7030 and stores these frames in the frame buffers 7042 and7044. While not illustrated, one or more additional integrated circuitsmay control the reading into and/or reading out of frames from the framebuffers 7042 and 7044 and operation of one or more other components ofthe display system 7000A, such as components of the image multiplexingsubsystem 7060, foveal-tracking beam-steering components 7080, and thelike. Reading into and/or out of the frame buffers 542 and 544 mayemploy dynamic addressing, for instance, where frames are over-rendered.The display system 7000A further comprises a read only memory (ROM) anda random access memory (RAM). The display system 7000A further comprisesa three-dimensional data base 7030 from which the GPU 7020 can accessthree-dimensional data of one or more scenes for rendering frames.

The CPU 7010 can include a high-FOV low-resolution render perspectivedetermination module 7012 and a low-FOV high-resolution renderperspective determination module 7014. In some embodiments, the userorientation determination module 7004 can be part of the CPU 7010.

The high-FOV low-resolution render perspective determination module 7012can include logic for mapping the data output by the user orientationdetermination module to the location in 3D space and the angle fromwhich high-FOV low-resolution images are to be perceived. That is, theCPU 7010 determines the perspective of a virtual camera fixed withrespect to the user's head at any given time based on the data receivedfrom the user orientation determination module 7004. Within the contextof the examples described above with reference to FIGS. 26A-26D and28A-28B, the high-FOV low-resolution render perspective determinationmodule 7012 may serve to monitor head position and orientation, asindicated by the user orientation determination module 7004, and controlthe position and orientation of at least the head-tracked virtual camerawithin render space accordingly.

The low-FOV high-resolution render perspective determination module 7014can include logic for mapping the data output by the user orientationdetermination module (e.g., data indicating the user's gaze and fovealpositioning) to the location in 3D space and the angle from whichlow-FOV high-resolution images are to be perceived. That is, the CPU7010 determines the perspective of a virtual camera fixed with respectto the user's fovea at any given time based on the data received fromthe user orientation determination module 7004. Within the context ofthe examples described above with reference to FIGS. 26A-26D and28A-28B, the low-FOV high-resolution render perspective determinationmodule 7014 may serve to monitor eye gaze, as indicated by the userorientation determination module 7004, and control the position andorientation of at least the fovea-tracked virtual camera within renderspace accordingly.

The display system 7000A can further include a graphics processing unit(GPU) 7020 and a database 7030. The database 7030 can store 3D virtualcontent. The GPU 7020 can access the 3D virtual content stored in thedatabase 7030 for rendering frames. The GPU 7020 can render frames ofvirtual content in low FOV and high resolution from the perspective ofthe virtual camera fixed with respect to the user's fovea (e.g.,fovea-tracked render perspective), as determined and provided as outputby the CPU 7010. The GPU 7020 can also render frames of virtual contentin high FOV and low resolution from the perspective of the virtualcamera fixed with respect to the user's head (e.g.,head-tracked/non-foveated perspective), as determined and provided asoutput by the CPU 7010. Further details discussing the creation,adjustment, and use of virtual cameras in rendering processes areprovided in U.S. patent application Ser. No. 15/274,823, entitled“METHODS AND SYSTEMS FOR DETECTING AND COMBINING STRUCTURAL FEATURES IN3D RECONSTRUCTION,” which is expressly incorporated herein by referencein its entirety for all purposes.

The high-FOV low-resolution rendered frames of virtual content can bestored in a high-FOV low-resolution rendered frame buffer 7042.Similarly, the low-FOV high-resolution rendered frames of virtualcontent can be stored in a low-FOV high-resolution rendered frame buffer7044. In some embodiments, the high-FOV low-resolution rendered framebuffer 7042 and the low-FOV high-resolution rendered frame buffer 7044can be part of the GPU 7020.

The display system 7000A can further include an image multiplexingsubsystem 7060 and an image multiplexing subsystem controller 7050communicatively coupled to the image multiplexing subsystem 7060. Theimage multiplexing subsystem 7060 can include an image source 7062 andmultiplexing components 7064 for multiplexing high-FOV low-resolutionimage frames and low-FOV high-resolution image frames, substantially asdescribed in further detail below with reference to FIGS. 30A-30B. Theimage source 7062 can include, for example, a light source incombination with fiber scanning components, liquid crystal on silicon(LCoS), MEMs scanning mirror, and the like. The multiplexing components7064 can include optical elements, such as polarization rotators,switchable optics, liquid crystal arrays, varifocal lenses, and thelike. The multiplexing components 7064 can be internal or external tothe image source 7062.

The image multiplexing subsystem controller 7050 is communicativelycoupled to the image multiplexing subsystem 7060, the high-FOVlow-resolution rendered frame buffer 7042, and the low-FOVhigh-resolution rendered frame buffer 7044. The control circuitry cansend control signals to the image source 562, so that appropriate imagecontent is presented from each render perspective, as discussed above.The image multiplexing subsystem controller 7050 can also control themultiplexing components 7064 in conjunction with the image source 7062in a manner so as to yield a multiplexed image stream.

The display system 7000A can further include foveal-trackingbeam-steering components 7080 and a foveal-tracking controller 7070communicatively and/or operatively coupled to foveal-trackingbeam-steering components 7080. The foveal-tracking controller 7070 canreceive output data from the CPU 7010 regarding the position of theuser's fovea (e.g., as determined by the low-FOV high-resolution renderperspective determination module 7014 and/or the user orientationdetermination module 7004), and use such data to control the position ofthe foveal-tracking beam-steering components 7080. The foveal-trackingbeam-steering components 7080 can serve to dynamically steer orotherwise direct low-FOV high-resolution portions of the multiplexedimage stream (produced by the image source 7062 and the multiplexingcomponents 7064) toward the user's fovea. Such low-FOV high-resolutionportions of the image stream may, for instance, represent virtualcontent as would be captured from the perspective of a fovea-trackedvirtual camera.

The display system 7000A can also include a storage medium for storingcomputer-readable instructions, databases, and other information usableby the CPU 7010, GPU 7020, and/or one or more other modules orcontrollers of the display system 7000A. The display system 7000A canfurther include input-output (I/O) interfaces, such as buttons, that auser may use for interaction with the display system. The display system7000A can also include a wireless antenna for wireless communicationwith another part of the display system 7000A, or with the Internet.

FIG. 29B illustrates schematically a cross-sectional view of an ARsystem 7000B according to some embodiments. The AR system 7000B canincorporate at least some of the components of the display system 7000Aas described above with reference to FIG. 29A, and can be fitted intoone of the displays 4052 in the wearable display device 4050 as shown inFIG. 25A according to some embodiments. For instance, the AR system7000B can include an image multiplexing subsystem 560, which can includean image source 7062 and one or more multiplexing components. Inaddition, the AR system 7000B can also include foveal-trackingbeam-steering components 7080, which in this example may anelectromechanical optical device, such as a MEMs scanning mirror. Muchlike the display system 7000A, the image multiplexing subsystem 7060 maybe communicatively and/or operatively coupled to an image multiplexingsubsystem controller, and the foveal-tracking beam-steering components7080 may be communicatively and/or operatively coupled to afoveal-tracking controller. The AR system 7000B can further include oneor more incoupling gratings (ICGs) 7007, and one or more eyepieces 7008.Each incoupling grating 7007 can be configured to couple the first lightbeam and the second light beam into a respective eyepiece 7008. Eacheyepiece 7008 can include outcoupling gratings for outcoupling the firstlight beam and the second light beam into a user's eye. The incouplinggratings 7007 and the eyepieces 7008 may be referred herein as a“viewing assembly.” It will be appreciated that the various incouplinggratings (ICG's) disclosed herein may correspond to the in-couplingoptical elements 700, 710, 720 of FIGS. 9A-9C.

FIGS. 30A-30B illustrate schematically a display system 8000 forprojecting images to an eye of a user according to some embodiments. Thedisplay system 8000 includes an image source 8010. The image source 8010can be configured to project a first light beam 8052 associated with afirst image stream, as shown in FIG. 30A, and project a second lightbeam 8054 associated with a second image stream, as shown in FIG. 30B.It should be noted that, the first light beam 8052 and the second lightbeam 8054 are depicted in FIGS. 30A-30B as schematic light rays, whichare not intended to represent accurate ray-traced rays. The first lightbeam 8052 can be angularly magnified to cover a wider FOV, resulting ina lower angular resolution image stream. The second light beam 8054 canhave a narrower FOV with a higher angular resolution, as discussed abovewith reference to FIGS. 26A-26F and 28A-28D.

The image source 8010 may include a liquid crystal on silicon (LCoS orLCOS) display (can also be referred to as a spatial light modulator), ascanning fiber, or a scanning mirror according to various embodiments.For example, the image source 8010 may include a scanning device thatscans an optical fiber in a predetermined pattern in response to controlsignals. The predetermined pattern can correspond to certain desiredimage shape, such as rectangle or circular shapes.

According to some embodiments, the first light beam 8052 associated withthe first image stream and the second light beam 8054 associated withthe second image stream can be multiplexed and output by the imagesource 8010 as composite light beams. For example, polarization-divisionmultiplexing, time-division multiplexing, wavelength-divisionmultiplexing, and the like, can be used for multiplexing the light beamsassociated with the first image stream and the light beams associatedwith the second image stream.

In embodiments where polarization-division multiplexing is used, thefirst light beam 8052 can be in a first polarization state, and thesecond light beam 8054 can be in a second polarization state differentfrom the first polarization state. For example, the first polarizationstate can be a linear polarization oriented in a first direction, andthe second polarization state can be a linear polarization oriented in asecond direction orthogonal to the first direction. In some otherembodiments, the first polarization state can be a left-handed circularpolarization, and the second polarization state can be a right-handedcircular polarization, or vice versa. The first light beam 8052 and thesecond light beam 8054 can be projected by the image source 8010simultaneously or sequentially.

The display system 8000 can further include a polarization beam splitter(PBS) 8030 configured to de-multiplex the first light beam 8052 from thesecond light beam 8054 according to some embodiments. The polarizationbeam splitter 8030 can be configured to reflect the first light beam8052 along a first optical path toward a viewing assembly as illustratedin FIG. 30A, and to transmit the second light beam 8054 along a secondoptical path as illustrated in FIG. 30B.

Alternatives to polarization beam splitter 8030 may also be used forde-multiplexing light beams. As an example, the beam splitters describedherein, including but not limited to polarization beam splitter 8030 ofFIGS. 30A and 30B, may be replaced or implemented with a switchablereflector, such as a liquid crystal switchable reflector. In embodimentswith such a switchable reflector, all other aspects disclosed hereinapply and may be similar, except that the polarization beam splitter isreplaced by the switchable reflector. As an example, a switchablereflector, such as switchable reflector 50042 of FIG. 53A, may switchbetween a reflective state and a transparent state in response tocontrol signals. By coordinating the switching of the switchablereflector, the switchable reflector may operate to de-multiplex lightbeams. As an example, the switchable reflector may be made reflective attimes when a first light beam is incident on the switchable reflectorand may be made transparent at times when a second light beam isincident on the switchable reflector, thus permitting de-multiplexing ofthe first and second light beams. In some embodiments, the switchablereflector may be positioned at an angle (e.g., a 45° angle) relative tothe light beams 8052, 8054. As a result, in a transmissive state, one ofthe light beams 8052, 8054 is transmitted through the switchablereflector; and in a reflective state, the other one of the light beams8054, 8052 is reflected such that it travels in a different directionaway from the switchable reflector than the light beam that wastransmitted through the reflector.

Referring to FIG. 30B, the display system 8000 can further include ascanning mirror 8060 positioned downstream from the polarization beamsplitter 8030 along the second optical path. The scanning mirror 8060 isconfigured to reflect the second light beam 8054 toward the viewingassembly to be projected to the user's eye. According to someembodiments, the scanning mirror 8060 can be controlled based on thefixation position of the user's eye for dynamically projecting thesecond image stream. For example, the scanning mirror 8060 can be inelectrical communication via control circuitry with an eye-gaze trackerthat tracks the user's eye movement. The control circuitry can send acontrol signal to tilt and/or translate the scanning mirror 8060 basedon the user's current fixation point, such that the second light beam8054 project the second image stream to a region determined to cover theuser's foveal vision. In some embodiments, the scanning mirror 8060 canbe a microelectromechanical systems (MEMS) scanner with two degrees offreedom (i.e., capable of being scanned in two independent angles).

In some other embodiments, instead of using a scanning mirror 8060, thedisplay system 8000 can use a fixed mirror. Controlling the position ofthe second image stream can be achieved by transversely displacing athird optical lens 8046 (see the description of the third optical lens8046 below). For example, the third optical lens 8046 can be displacedup and down as indicated by the arrow, as well as in and out of thepage, to shift the position of the second image stream in twodimensions.

In some embodiments, the display system 8000 can further include apolarization rotator 8022 positioned between the polarization beamsplitter 8030 and the scanning mirror 8060. The polarization rotator8022 can be configured to rotate the polarization of the second lightbeam 8054, so that the second light beam can have approximately the samepolarization as that of the first light beam 8052 as they enter theviewing assembly. The polarization rotator 8022 can include, forexample, a half-wave plate.

In some embodiments, the display system 8000 can further include a firstrelay lens assembly for the first optical path, and a second relay lensassembly for the second optical path. The first relay lens assembly caninclude a first optical lens 8042 disposed between the image source 8010and the polarization beam splitter 8030, and a second optical lens 8044disposed downstream from the polarization beam splitter 8030 along thefirst optical path. The second relay lens assembly can include the firstoptical lens 8042, and a third optical lens 8046 disposed downstreamfrom the polarization beam splitter 8030 along the second optical path.

FIG. 30C illustrates schematically a cross-sectional view of anaugmented reality (AR) system according to some embodiments. The ARsystem can be fitted into one of the displays 4052 in the wearabledisplay device 4050 as shown in FIG. 25A according to some embodiments.The AR system can include a light projector 8000 for projecting a firstlight beam associated with a first image stream and a second light beamassociated with a second image stream. The light projector 8000 can besimilar to the display system illustrated in FIGS. 30A-30B. The ARsystem can further include one or more incoupling gratings (ICGs) 8070,and one or more eyepieces 8080. Each incoupling grating 8070 can beconfigured to couple the first light beam and the second light beam intoa respective eyepiece 8080. Each eyepiece 8080 can include outcouplinggratings for outcoupling the first light beam and the second light beaminto a user's eye. The incoupling gratings 8070 and the eyepieces 8080may be referred herein as a “viewing assembly.”

FIG. 30D shows a simplified block diagram of a display system accordingto some embodiments. The display system can include an image source8010, and a scanning mirror 8060, substantially as described above withreference to FIGS. 30A-30C. The display system can also include aneye-gaze tracker 8071 and control circuitry 8081. The control circuitry8081 can be communicatively coupled to the image source 8010, thescanning mirror 8060, and the eye-gaze tracker 8071. The controlcircuitry 8081 can send control signals to tilt and/or translate thescanning mirror 8060 based on the user's current fixation point asdetermined by the eye-gaze tracker 8071, so that the second light beam8054 project the second image stream to a region determined to cover theuser's foveal vision. The control circuitry 8081 can also send controlsignals to the image source 8010, so that appropriate image content ispresented in the first image stream and the second image stream, asdiscussed above. The display system can also include a centralprocessing unit (CPU) 8096, a graphics processing unit (GPU) 8098, astorage medium 8090 for storing computer-readable instructions,databases, and other information usable by the control circuitry 8081,the CPU 8096, and the GPU 8098. The display system can further includeinput-output (I/O) interfaces 8092, such as buttons, that a user may usefor interaction with the display system. The display system can alsoinclude a wireless antenna 8094 for wireless communication with anotherpart of the display system, or with the Internet. The display system canalso include other sensors, such as cameras.

FIG. 31A illustrates schematically the operating principles of the firstrelay lens assembly according to some embodiments. The first relay lensassembly can operate in a manner similar to a telescope. A collimatedfirst light beam 8052 associated with the first image stream is incidenton the first optical lens 8042 at an angle of incidence θ_(A), and isfocused by the first optical lens 8042 to a real image point P₀ locatedapproximately at a focal plane of the first optical lens 8042. The realimage point P₀ is also located approximately at a focal plane of thesecond optical lens 8044. Thus, the first light beam 8052 emitted fromthe real image point P₀ is collimated by the second optical lens 80044and exits from the second optical lens 8044 at an angle of transmittanceθ_(B).

The ratio of θ_(B) and θ_(A) can give rise to a first angularmagnification M₁, where

${M_{1} = \frac{\theta_{B}}{\theta_{A}}}.$The magnitude of first angular magnification M₁ can be approximatelyequal to the ratio of the focal length of the first optical lens 8042f_(A) and the focal length of the second optical lens 8044 f_(B). Thus

${M_{1} \approx \frac{f_{A}}{f_{B}}}.$In some embodiments, the first relay lens assembly is configured suchthat the magnitude of the first angular magnification M₁ is greater thanone, e.g., by having f_(A)>f_(B). Therefore, referring again to FIG.30A, the collimated first light beam 8052 associated with the firstimage stream can be angularly magnified by the first relay lens assemblyas it exits the second optical lens 8044, which is then projected to aviewing assembly for presenting the first image stream with a firstfield of view FOV₁ that is relatively wide.

FIG. 31B illustrates schematically the operating principles of thesecond relay lens assembly according to some embodiments. The secondrelay lens assembly can also operate in a similar manner as a telescope.A collimated second light beam 8054 associated with the second imagestream is incident on the first optical lens 8042 at an angle ofincidence θ_(A), and is focused by the first optical lens 8042 to a realimage point P₀ located approximately at a focal plane of the firstoptical lens 8042. The real image point P₀ is also located approximatelyat a focal plane of the third optical lens 8046. Thus, the second lightbeam 8054 emitted from the real image point P₀ is collimated by thethird optical lens 8046 and exits from the third optical lens 8046 at anangle of transmittance θ_(C).

The ratio of θ_(C) and θ_(A) can give rise to a second angularmagnification M₂, where

$M_{2} = {\frac{\theta_{C}}{\theta_{A}}.}$The magnitude of second angular magnification M₂ can be approximatelyequal to the ratio of the focal length of the first optical lens 8042f_(A) and the focal length of the third optical lens 644 f_(C). Thus

${M_{2} \approx \frac{f_{A}}{fc}}.$The second lens assembly can be configured such that the magnitude ofthe second angular magnification M₂ is less than the first angularmagnification M₁. In some embodiments, the second angular magnificationM₂ can have a value of unity (i.e., no magnification) or less than one(i.e., demagnification), e.g., by having f_(A)≤f_(C). Therefore,referring again to FIG. 30B, the collimated second light beam 8054associated with the second image stream can have a second field of viewFOV₂ as it exits the third optical lens 8046, the second field of viewFOV₂ being less than the first field of view FOV₁ of the first lightbeam 8052 associated with the first image stream.

Note in FIG. 31A that the collimated first light beam 8052 has aninitial beam width w_(A) as it is incident on the first optical lens8042, and a final beam width w_(B) as it exits the second optical lens8044, where the final beam width w_(C) is narrower than the initial beamwidth w_(A). Note also in FIG. 31B that the collimated second light beam8054 has an initial beam width w_(A) as it is incident on the firstoptical lens 8042, and a final beam width w_(C) as it exits the thirdoptical lens 8046, where the final bam width w_(C) is about the same asthe initial beam width w_(A). In other words, the final beam width w_(C)of the second light beam 8054 is wider than the final beam width w_(B)of the first light beam 8052. A wider beam width would result in asharper angular resolution perceived by the eye. This can be explainedby Gaussian beam physics, where a collimated beam with a wider beamwaist has lower angular divergence over propagation to infinity.Therefore, increasing the FOV can reduce the beam width, and hence canreduce the angular resolution, which is consistent with the Lagrangeinvariant.

In some embodiments, the first angular magnification M₁ can have amagnitude of about 3, and the second angular magnification M₂ can have amagnitude of about unity. Referring to FIGS. 30A-30B, assume that thecollimated first light beam 8052 associated with the first image streamand the collimated second light beam 8054 associated with the secondimage stream have the same initial FOV of about 20 degrees as projectedby the image source 8010. The collimated first light beam 8052 exitingthe second optical lens 644 can have a first field of view FOV₁ of about60 degrees, whereas the collimated second light beam 654 exiting thethird optical lens 8046 can have a second field of view FOV₂ of about 20degrees. In some embodiments, the first FOV can range from about 30degrees to about 90 degrees; and the second FOV can range from about 10degrees to about 30 degrees.

As illustrated in FIGS. 28C-28D, the second image stream 6020 can be ahigh resolution version of a portion of the first image stream 6010 andis overlaid on and properly aligned with respect to the wide FOV and lowresolution first image stream 6010. The content of the second imagestream 6020 changes as the second image stream shifts relative to thefirst image stream 6010, so that the content of the second image stream6020 corresponds to the portion of the first image stream 6010 overlaidby the second image stream 6020. Because the second image stream 6020persistently covers the user's foveal vision, the user can perceive thecombination of the first image stream 6010 and the second image stream6020 as a composite image stream that has both a wide FOV and a highresolution.

FIGS. 31C-31D illustrate schematically a display system 10000 accordingto some other embodiments. The display system 10000 includes an imagesource 9010 and a beam splitter 9030. The image source 9010 can providea first light beam 8052 associated with a first image stream and asecond light beam 8054 associated with a second image stream. The firstlight beam 8052 and the second light beam 8054 can be time-divisionmultiplexed, polarization-division multiplexed, wavelength-divisionmultiplexed, or the like. The beam splitter 9030 can serve as ade-multiplexer to separate the first light beam 8052 and the secondlight beam 8054 toward a first optical path and a second optical path,as depicted in FIGS. 31C and 31D, respectively.

The display system 10000 can also include a first optical lens 9042 anda second optical lens 9044 disposed downstream from the beam splitter9030 along the first optical path. The combination of the first opticallens 9042 and the second optical lens 9044 can serve as a first relaylens assembly for the first light beam 8052. In some embodiments, thefirst relay lens assembly can provide an angular magnification for thefirst light beam 8052 that is greater than one, as described above inrelation to FIG. 31A.

The display system 10000 can also include a third optical lens 9045 anda fourth optical lens 9046 disposed downstream from the beam splitter9030 along the second optical path. The combination of the third opticallens 9045 and the fourth optical lens 9046 can serve as a second relaylens assembly for the second light beam 8054. In some embodiments, thesecond relay lens assembly can provide an angular magnification for thesecond light beam 8054 that is substantially unity or less than one, asdescribed above in relation to FIG. 31B.

The display system 10000 can also include a scanning mirror 9060positioned downstream from the second relay lens assembly along thesecond optical path. The scanning mirror 9060 is configured to reflectthe second light beam 8054 toward a viewing assembly to be projected tothe user's eye. According to some embodiments, the scanning mirror 9060can be controlled based on the fixation position of the user's eye fordynamically projecting the second image stream.

The display system 10000 can also include a fifth optical lens 9047 anda sixth optical lens 9048 disposed downstream from scanning mirror 9060along the second optical path. The combination of the fifth optical lens9047 and the sixth optical lens 9048 can serve as a third relay lensassembly for the second light beam 8054. In some embodiments, the thirdrelay lens assembly can provide an angular magnification for the secondlight beam 8054 that is substantially unity or less than one, asdescribed above in relation to FIG. 31B.

In some embodiments, the display system 10000 can also include apolarizer 9080 and a switching polarization rotator 9090. The imagesource 9010 can provide an unpolarized first light beam 8052 and anunpolarized second light beam 8054, which are time-division multiplexed.The first light beam 652 and the second light beam 654 may becomepolarized after passing through the polarizer 9080. The switchingpolarization rotator 9090 can be operated in synchronization with thetime-division multiplexing of the first light beam 8052 and the secondlight beam 8054. For example, the switching polarization rotator 9090can be operated such that the polarization of the first light beam 8052is unchanged after passing through the switching rotator 9090, whereasthe polarization of the second light beam 8054 is rotated by 90 degreesafter passing through the switching polarization rotator 9090, or viceversa. Therefore, the first light beam 8052 can be reflected by thepolarization beam splitter 9030 along the first optical path asillustrated in FIG. 31C, and the second light beam 8054 can betransmitted by the polarization beam splitter 9030 along the secondoptical path, as illustrated in FIG. 31D.

FIGS. 32A-32C illustrate schematically a display system 10000 accordingto some other embodiments. In some examples, one or more components ofdisplay system 10000 may be the same as or similar to one or morecomponents of the display system as described above with reference toFIGS. 31C-31D. The display system 10000 includes an image source 10010,a beam splitter 10030, a first optical lens 10042, a second optical lens10044, a third optical lens 10045, a fourth optical lens 10046, a fifthoptical lens 10047, a sixth optical lens 10048, a scanning mirror 10060,a polarizer 10080, a switching polarization rotator 10090 that, in someexamples, may be the same as or similar to elements 9010, 9030, 9042,9044, 9045, 9046, 9047, 9048, 9060, 9080, and 9090, respectively, of thedisplay system as described above with reference to FIGS. 31C-31D.

More specifically, FIGS. 32A-32C illustrate a display system 10000 ineach of three different stages. In each of the three stages, the imagesource 10010 can output a range of angular light field componentsrepresentative of virtual content as would be captured from theperspective of a head-tracked virtual camera and a range of angularlight field components representative of virtual content as would becaptured from the perspective of a fovea-tracked virtual camera. The twosets of angular light field components may, for instance, betime-division multiplexed, polarization-division multiplexed,wavelength-division multiplexed, or the like. As such, the angular lightfield components associated with the head-tracked virtual camera can bediverted upward by the polarization beam splitter 10030 along a firstoptical path through the first and second optical lenses 10042 and10044, and the angular light field components associated with thefovea-tracked virtual camera can pass through the polarization beamsplitter 10030 along a second optical path through third and fourthoptical lenses 10045 and 10046 toward the scanning mirror 10060 andreflected upward through fifth and sixth optical lenses 10047 and 10048.

The virtual content represented by the angular light field componentsassociated with the head-tracked virtual camera may be rendered upstreamfrom the image source 10010 at a relatively low resolution, while thevirtual content represented by the angular light field componentsassociated with the fovea-tracked virtual camera may be renderedupstream from the image source 10010 at a relatively high resolution.And, as shown in FIGS. 32A-32C, the display system 10000 may beconfigured to output the angular light field components associated withthe head-tracked render perspective and the angular light fieldcomponents associated with the fovea-tracked render perspective as highFOV and low FOV light fields, respectively. In each of FIGS. 32A-32C,the light field components that propagate along the first optical pathare output by the display system 10000 as a relatively wide cone oflight 10052.

In the stage depicted in FIG. 32A, the scanning mirror 10060 is in afirst position. As such, it can be seen that the light field componentsthat pass through the polarization beam splitter 10030 and propagatealong the second optical path are output by the display system 10000 asa relatively narrow cone of light 10054A spanning a substantiallycentral region of angular space. Within the context of the examplesdescribed above with reference to FIGS. 28A-28B, the display system10000 could, for instance, place the scanning mirror 10060 in the firstposition shown in FIG. 32A when the user's eye is oriented in a mannersimilar to that of the viewer's eye 210 in FIG. 28A. In this way, thelight components 10054A may represent virtual content in a relativelycentralized region of render space, such as virtual object 6012. Furtherto the examples of FIGS. 28A-28B, the relatively wide cone of light10052 may, for instance, include virtual content in off-centered regionsof render space, such as virtual objects 6011 and 6013. In someexamples, the relatively wide cone of light 10052 may further includelight components that represent the same virtual content as isrepresented by the light components 10054A, but in lower resolution.

In the stage depicted in FIG. 32B, the scanning mirror 10060 is in asecond position different from the first position. As such, it can beseen that the light field components that pass through the polarizationbeam splitter 10030 and propagate along the second optical path areoutput by the display system 10000 as a relatively narrow cone of light10054B spanning one substantially off-centered region of angular space.Within the context of the examples described above with reference toFIGS. 28A-28B, the display system 10000 could, for instance, place thescanning mirror 10060 in the second position shown in FIG. 32B when theuser's eye is oriented in a manner similar to that of the viewer's eye210 while the viewer is looking at virtual object 6011. In this way, thelight components 10054B may represent virtual content in one relativelyoff-centered region of render space, such as virtual object 6011.Further to the examples of FIGS. 28A-28B, the relatively wide cone oflight 10052 may, for instance, include virtual content in the otheroff-centered region of render space, such as virtual object 6013, aswell as virtual content in the centralized region of render space, suchas virtual object 6012. In some examples, the relatively wide cone oflight 10052 may further include light components that represent the samevirtual content as is represented by the light components 10054B, but inlower resolution.

In the stage depicted in FIG. 32C, the scanning mirror 10060 is in athird position different from the first and second positions. As such,it can be seen that the light field components that pass through thepolarization beam splitter 10030 and propagate along the second opticalpath are output by the display system 10000 as a relatively narrow coneof light 10054C spanning another, different substantially off-centeredregion of angular space. Within the context of the examples describedabove with reference to FIGS. 28A-28B, the display system 10000 could,for instance, place the scanning mirror 10060 in the second positionshown in FIG. 32C when the user's eye is oriented in a manner similar tothat of the viewer's eye 210 in FIG. 28B. In this way, the lightcomponents 10054C may represent virtual content in the other relativelyoff-centered region of render space, such as virtual object 6013.Further to the examples of FIGS. 28A-28B, the relatively wide cone oflight 10052 may, for instance, include virtual content in theoff-centered region of render space described above with reference toFIG. 32B, such as virtual object 6011, as well as virtual content in thecentralized region of render space, such as virtual object 6012. In someexamples, the relatively wide cone of light 10052 may further includelight components that represent the same virtual content as isrepresented by the light components 10054C, but in lower resolution.

FIGS. 33A-33B illustrate schematically a display system 11000 forpresenting a first image stream and second image stream, wheretime-division multiplexing is used for multiplexing the first light beam8052 associated with the first image stream and the second light beam8054 associated with the second image stream, according to someembodiments. The display system 11000 is similar to the display system8000. The image source 11010 can be configured to provide time-divisionmultiplexed first light beam 8052 and second light beam 8054. The firstlight beam 8052 and the second light beam 8054 can be in the samepolarization state as output from the image source 8010. It should benoted that the first light beam 8052 and the second light beam 8054 aredepicted in FIGS. 33A-33B as schematic light rays, which are notintended to represent accurate ray-traced rays.

The display system 11000 can further include a switching polarizationrotator 11020, whose operation can be synchronized with thetime-division multiplexing of the first light beam 8052 and the secondlight beam 8054. For example, the switching polarization rotator 11020can be operated such that the polarization of the first light beam 8052is unchanged after passing through the switching rotator 11020, whereasthe polarization of the second light beam 8054 is rotated by 90 degreesafter passing through the switching polarization rotator 11020, or viceversa. Therefore, the first light beam 8052 can be reflected by thepolarization beam splitter 8030 along the first optical path asillustrated in FIG. 33A, and the second light beam 8054 can betransmitted by the polarization beam splitter 8030 along the secondoptical path, as illustrated in FIG. 33B.

In some other embodiments, the switching polarization rotator 11020 canbe part of the image source 11010. In such cases, the first light beam8052 and second light beam 8054 would be emitted sequentially and thefirst light beam 8052 projected from the image source 8010 would bepolarized in a first direction, and the second light beam 8054 projectedfrom the image source 8010 would be polarized in a second direction.

According to some embodiments, in cases where the first light beam 8052associated with the first image stream and the second light beam 8054associated with the second image stream are time-division multiplexed, aswitchable mirror can be used in place of the polarization beam splitter8030 shown in FIGS. 30A-30B, 31C-31D, and 33A-33B. The switching of theswitchable mirror can be synchronized with the time-divisionmultiplexing of the first light beam 8052 and the second light beam8054. For example, the switchable mirror can be switched to a firststate for the first light beam 8052 so that it operates as a mirrorreflecting the first light beam 8052 along the first optical path asillustrated in FIGS. 30A, 31C, and 33A, and be switched to a secondstate for the second light beam 8054 so that it operates as atransparent optical element transmitting the second light beam 8054along the second optical path as illustrated in FIGS. 30B, 31D, and 33B.

According to some embodiments, wavelength-division multiplexing can beused for multiplexing the first light beam associated with the firstimage stream and the second light beam associated with the second imagestream. For example, the first light beam can be composed of light in afirst set of wavelength ranges in red, green, and blue, and the secondlight beam can be composed of light in a second set of wavelength rangesin red, green, and blue light. The two sets of wavelength ranges can beshifted with respect to each other, but the composite of the second setof wavelength ranges produces a white light that is substantially thesame as the white light produced by the composite of the first set ofwavelength ranges.

In cases where wavelength-division multiplexing is used, a displaysystem can include a dichroic beam splitter that takes the place of thepolarization beam splitter to separate the first light beam associatedwith the first image stream and the second light beam associated withthe second image stream. For example, the dichroic beam splitter can beconfigured to have a high reflectance value and a low transmittancevalue for the first set of wavelength ranges, and a low reflectancevalue and a high transmittance value for the second set of wavelengthranges. In some embodiments, the first light beam and the second lightbeam can be projected concurrently without the need for a switchablepolarization rotator.

FIGS. 34A-34B illustrate schematically a display system 12000 accordingto some other embodiments. The display system 12000 includes an imagesource 12010. The image source 12010 can be configured to project firstlight beam 12052 associated with a first image stream as illustrated inFIG. 34A, and second light beam 12054 associated with a second imagestream as illustrated in FIG. 34B. The first image stream can be a wideFOV and low resolution image stream, and the second image stream can bea narrow FOV and high resolution image stream, as discussed above withreference to FIGS. 26E-26F. The first light beam 12052 and the secondlight beam 12054 can be multiplexed using, for example,polarization-division multiplexing, time-division multiplexing,wavelength-division multiplexing, and the like. In FIGS. 34A-34B, thefirst light beam 12052 and the second light beam 12054 are depicted asschematic light rays, which are not intended to represent accurateray-traced rays.

The display system 12000 can further include a beam splitter 12030configured to de-multiplex the first light beam 12052 and the secondlight beam 12054 according to some embodiments. For example, the beamsplitter 12030 can be a polarization beam splitter (PBS) or a dichroicbeam splitter. The beam splitter 12030 can be configured to reflect thefirst light beam 12052 along a first optical path as illustrated in FIG.34A, and to transmit the second light beam 12054 along a second opticalpath as illustrated in FIG. 34B.

The display system 12000 can further include a switchable opticalelement 12040. Although the switchable optical element 12040 isillustrated as a single element, it can include a pair of sub switchableoptical elements that functions as a switchable relay lens assembly.Each sub switchable optical element can be switched to a first statesuch that it operates as an optical lens with a first optical power, orbe switched to a second state such that it operates as an optical lenswith a second optical power different than the first optical power. Assuch, the switchable optical element 12040 can provide a first angularmagnification when the sub switchable optical elements are switched tothe first state, as illustrated in FIG. 34A, and a second angularmagnification different from the first angular magnification when thesub switchable optical elements are switched to the first state, asillustrated in FIG. 34B.

Each sub switchable optical element can take many forms, including e.g.,liquid crystal varifocal lenses, tunable diffractive lenses, ordeformable lenses. In general, any lens that could be configured tochange shape or configuration to adjust its optical power could beapplied. In some embodiments, each sub switchable optical element can bea multifocal birefringent lens that has a first optical power for alight with a first polarization and a second optical power substantiallydifferent from the first optical power for light with a secondpolarization. For example, a multifocal birefringent lenses can comprisea polymer that has been made birefringent by an orienting process bystretching the polymer under defined conditions, such that the polymerexhibits an ordinary refractive index no and an extraordinary refractiveindex n_(e).

In cases where the first light beam 12052 and the second light beam12054 are time-division multiplexed, the switching of the switchableoptical element 12040 can be synchronized with the time-divisionmultiplexing of the first light beam 12052 and the second light beam12054, so that each sub switchable optical element operates as anoptical lens with the first optical power for the first light beam 12052as illustrated in FIG. 34A, and operates as an optical lens with thesecond optical power for the second light beam 12054 as illustrated inFIG. 34B. Therefore, the first light beam 12052 associated with thefirst image stream can be angularly magnified by the switchable opticalelement 12040 as they exit the switchable optical element 12040, and canbe subsequently projected to a viewing assembly for presenting the firstimage stream with a first field of view FOV, that is relatively wide.

The display system 12000 can further include a first mirror 12060positioned downstream from the beam splitter 12030 along the secondoptical path as illustrated in FIG. 34B. The first mirror 12060 canreflect the second light beam 12054 back toward the beam splitter 12030,which can be subsequently reflect by the beam splitter 12030 towards asecond mirror 12070.

The second mirror 12070 is positioned below the beam splitter 12030 asillustrated in FIG. 34B. The second mirror 12070 can reflect the secondlight beam 12054 back toward the beam splitter 12030, which can besubsequently transmitted by the beam splitter 12030 toward theswitchable optical element 12040. As described above, each subswitchable optical element can be switched to the second state so thatit can operate as an optical lens with the second optical power for thesecond light beam 12054. The second optical power can be less than thefirst optical power associated with the first state, or be substantiallyzero or negative. Therefore, the second light beam 12054 can beangularly magnified by an amount less than the first light beam 12052,or be not magnified or be demagnified as they exit the switchableoptical element 12040. Thus, the second light beam 12054 can besubsequently projected to the viewing assembly for presenting the secondimage stream with a second field of view FOV₂ that is relatively narrow.

In some embodiments, the second mirror 12070 can be configured as atwo-dimensional (2D) scanning mirror (i.e., a scanning mirror with twodegrees of rotational freedom), such as a 2D MEMS scanner, that can betilted in two directions as illustrated in FIG. 34B. The tilting of thesecond mirror 12070 can be controlled based on the fixation position ofthe user's eye, such that the second light beam 12054 can project thesecond image stream at the user's foveal vision. In some otherembodiments, the second mirror 12070 can be a fixed mirror, and thefirst mirror 12060 can be a 2D scanning mirror. In some furtherembodiments, the first mirror can be a one-dimensional (1D) scanningmirror (i.e., a scanning mirror with one degree of rotational freedom)that can be tilted in a first direction, and the second mirror can be a1D scanning mirror that can be tilted in a second direction.

FIG. 35 illustrates schematically a display system 13000 according tosome other embodiments. The display system 13000 includes an imagesource 13010. The image source 13010 can be configured to provide afirst light beam associated with a first image stream in right-handedcircular polarization (RHCP) and a second light beam associated with asecond image stream in left-handed circular polarization (LHCP) (or viceversa).

The display system 13000 can further include a beam splitter 13030configured to de-multiplex the first light beam and the second lightbeam. For example, the beam splitter 13030 can comprise a liquid crystalmaterial that reflects the right-handed circularly polarized first lightbeam and transmits the left-handed circularly polarized second lightbeam.

The display system 13000 can further include a first switchable opticalelement 13042 and a second switchable optical element 13044, thecombination of which can serve as a relay lens assembly. Each of thefirst switchable optical element 13042 and the second switchable opticalelement 13044 can comprise a liquid crystal material such that it has afirst focal length f_(RHCP) for right-handed circular polarized lightand a second focal length f_(LHCP) for left-handed circularly polarizedlight. Therefore, the combination of the first switchable opticalelement 13042 and the second switchable optical element 13044 canprovide a first angular magnification to the first light beam, and asecond angular magnification to the second light beam that is differentfrom the first angular magnification. For example, the first angularmagnification can be greater than one, and the second angularmagnification can equal to unity or less than one.

FIG. 36 illustrates schematically an augmented reality near-eye displaysystem 14000 according to some embodiments. FIG. 36 shows a portion ofthe display systems 14000 for one eye 210. In practice a second suchsystem would be provided for a user's other eye. Two such systems areincorporated in augmented reality glasses according to embodiments.Referring to FIG. 36 , a red laser diode 14002 is optically coupledthrough a red laser collimating lens 14004 into a red light input face14006 of a Red-Green-Blue (RGB) dichroic combiner cube 14008. A greenlaser diode 14010 is optically coupled through a green laser collimatinglens 14012 into a green light input face 14014 of the RGB dichroiccombiner cube 14008. Similarly, a blue laser diode 14016 is opticallycoupled through a blue laser collimating lens 14018 into a blue lightinput face 14020 of the RGB dichroic combiner cube 14008. The RGBdichroic combiner cube 14008 has an output face 14022. The RGB dichroiccombiner cube 14008 includes a red reflecting dichroic mirror (shortwavelength pass mirror) 14024 set at 45 degrees so as to reflect lightfrom the red laser diode 14002 through the output face 14022. The RGBdichroic combiner cube 14008 also includes blue reflecting dichroicmirror (long wavelength pass) 14026 set at 135 degrees (perpendicular tored reflecting dichroic mirror 14024) so as to reflect light from theblue laser diode 14016 to the output face 14022. Light from the greenlaser diode 14010 passes through (is transmitted by) the red reflectingdichroic mirror 14024 and the blue reflecting dichoric mirror 14026 tothe output face 14022. The red reflecting dichroic mirror 14024 and theblue reflecting dichroic mirror 14026 can be implemented as thin filmoptical interference films.

The red, green, and blue laser diodes 14002, 14010, 14016 are separatelymodulated with red, blue and green color channel image information. Acycle including a first period in which image information to be directedto the fovea of a user's retina is output and a subsequent period inwhich image information to be directed to a larger portion of the user'sretina is repeated sequentially. There can be some angular overlapbetween image information directed to user's retina in the first periodand the image information directed to the user's retina during thesubsequent period of the cycle. In other words, certain portions of theuser's eye may receive light during both periods. Rather than trying toachieve a sharp boundary, overlapping boundaries characterized by atapering intensity may be used. The optical arrangement to achieve theaforementioned functionality will be described below.

The dichroic combiner cube 14008 outputs a collimated beam 14028 thatincludes red, blue and green components. The collimated beam 14028 isincident on a first two degree of freedom image scanning mirror 14030.The image scanning mirror 14030 has two degrees of freedom of rotationand can be oriented to angles within a predetermined angular range. Eachorientation of the image scanning mirror 14030 effectively correspondsto angular coordinates in an image space. The orientation of the imagescanning mirror 14030 is scanned in coordination with modulation of thered, green and blue laser diodes 14002, 14010, 14016 based on imageinformation so as to present an image, ultimately, to a user's eye.

Light deflected by the image scanning mirror 14030 is coupled through afirst relay lens element 14032 to a polarization rotation switch 14034.Alternatively, the polarization rotation switch could be located closerto the laser diodes 14002, 14010, 14016. The polarization rotationswitch 14034 is electrically controlled by electronics (not shown inFIG. 36 ). The polarization rotation switch 14034 can be implemented asa liquid crystal polarization rotation switch. The polarization rotationswitch 14034 receives light of a specific linear polarization that isoutput by the laser diodes 14002, 14010, 14016 and transferred throughthe collimating lenses 14004, 14012, 14018 and the RGB dichroic combinercube 14008 without altering the polarization. The polarization rotationswitch 14034 under the control of external electrical signals eitherpasses the incoming light without altering its polarization or rotatesthe polarization of the light by 90 degrees.

Light exiting the polarization rotation switch 14034 is coupled to apolarization beam splitter (PBS) 14036. The PBS 14036 has embeddedtherein a polarization selective reflector 14038 arranged diagonallyacross the PBS 14036. The polarization selective reflector 14038 can beof the type including an array of parallel metal conductive lines (notvisible in FIG. 36 ). Light polarized (i.e., have an electric fielddirection) parallel to the metal conductive lines is reflected and lightpolarized perpendicular to the conductive metal lines is transmitted. Inthe case of the embodiment shown in FIG. 36 it is assumed that theconductive metal lines are oriented perpendicular to the plane of thedrawing sheet. With such an orientation the polarization selectivereflector 14038 will reflect S-polarized light and transmit P-polarizedlight.

Considering first the case in which the polarization rotation switch14034 is in a state that outputs P-polarized light, such P-polarizedlight will pass through the polarization selective reflector 14038 andthrough the PBS 14036 entirely reaching a first quarter wave plate (QWP)14040. The first QWP 14040 is oriented so as to convert P-polarizedlight to right hand circularly polarized (RHCP) light. (Alternativelythe first QWP could have been oriented so as to convert P-polarizedlight to LHCP, in which changes to other components described below willalso be made as will be apparent after considering the remainingdescription of FIG. 36 .) After passing through the first QWP 14040light will reach a second relay lens element 14042. The first relay lenselement 14032 and the second relay lens element 14042 for a unitymagnification afocal compound lens. Note that the image scanning mirror14030 is spaced from the first relay lens element 14032 by a distanceequal to the focal length of the first relay lens element 14032. Thesecond relay lens element 14032 will recollimate the light (the lighthaving been initially collimated by collimating lenses 14004, 14012,14018). Note also that light propagating from the second relay lenselement 14042 will cross an optical axis OA near a point P1 that isspaced from the second relay lens element 14042 by the focal length ofthe second relay lens element 14042. In the embodiment shown in FIG. 36the first relay lens element 14032 and the second relay lens element14042 have the same focal length.

After exiting the second relay lens element 14042 the light will beincident on a first group positive refractive lens 14044 of a firstgroup 14046 of a dual magnification afocal magnifier 14048. In additionto the first group positive refractive lens 14044, the first group 14046also includes a first group geometric phase lens 14050. After passingthrough the first group geometric phase lens 14050, the light passesthrough a second group 14052 that includes a second group positiverefractive lens 14054 and a second group geometric phase lens 14056. Thegeometric phase lenses 14050, 14056 include patternwise aligned liquidcrystal material. Geometric phase lenses (also known as “polarizationdirected flat lenses”) are available from Edmund Optics of Barrington,N.J. The geometric phase lenses 14050, 14056 have the property that theyare positive lenses for circularly polarized light that has a handedness(RH or LH) that matches their handedness and are negative lenses forcircularly polarized light of opposite handedness. Geometric phaselenses also have the property that in transmitting light they reversethe handedness of circularly polarized light. In the embodiment shown inFIG. 36 , the geometric phase lenses 14050, 14056 are right handed. Itshould be noted that this system could be modified to accommodate usewith left handed geometric phase lenses.

In operation when RHCP light is passed through the first group 14046,the first group geometric phase lens 14050 will act as a negative lens,so that the positive optical power of the first group 14046 will be lessthan the positive optical power of the first group refractive lens 14044alone and the first group 14046 will have focal length about equal to adistance to point F_(RHCP) indicated in FIG. 36 from a principle planeof the first group 14046. Propagating through the first group geometricphase lens 14050 will convert the light to the left handed circularlypolarized (LHCP) state. For light of the LHCP state the second groupgeometric phase lens 14056 will have positive refractive power, andtherefore the positive refractive power of the second group 14052 willbe greater than the positive refractive power of the second grouppositive refractive lens 14054 alone. In this case a focal length of thesecond group 14052 will also equal a distance from the principle planeof the second group 14052 to the point F_(RHCP), with the subscript“RHCP” referring to the polarization state of the light entering themagnifier 14048. Because the point F_(RHCP) is closer to the secondgroup 14052 than the first group 14046, the dual magnification afocalmagnifier 14048 will be a magnifier (have a magnification greaterthan 1) for RHCP light received from the second relay lens element14042.

Now considering a second case in which the polarization rotation switch14034 is in a state that outputs S-polarized light, such S-polarizedlight is reflected by the polarization selective reflector 14038nominally 90 degrees and then passes through a second QWP 14058 andthereafter passes through a third relay lens element 14060 whichdeflects the light toward a fixed mirror 14062. Note that forS-polarized light the first relay lens element 14032 in combination withthe third relay lens element 14060 form a unity magnification afocalrelay. The fixed mirror 14062 reflects the light back through thirdrelay lens element 14060 and second QWP 14058 changing the sign but notthe absolute value of the angle of the light beam with respect to theoptical axis OA. After the first pass through the second QWP 14058 theS-Polarized light is converted to circularly polarized light of aparticular handedness (which can be chosen to be either RHCP or LHCP bychoosing the orientation of the fast and slow axes of the second QWP14058). Upon reflection by the fixed mirror 14062 the handedness of thecircularly polarized light is reversed. Upon the second pass through thesecond QWP the circularly polarized light which was S-polarized isconverted (temporarily) to P-polarized light which then passes throughthe polarization selective reflector 14038.

After passing through the polarization selective reflector 14038, thelight passes through a third QWP 14064 and a fourth relay lens element14066 and is directed to a fovea tracking mirror 14068. In the system14000, because the image scanning mirror 14030, the fixed mirror 14060and the fovea tracking mirror 14068 are spaced from respectively fromthe relay lens elements 14032, 14066, 14060 by the focal length of therelay lens element 14032, 14066, 14060 and the QWPs 14040, 14058, 14064are positioned after the relay lens elements 14032, 14042, 14060, 14066the angle of light incidence on the QWPs 14040, 14058, 14064 isrelatively low which leads to improved performance of the QWPs 14040,14058, 14064. According to an alternative embodiment, rather than havinga single fovea tracking mirror 1268 that tracks two angular degrees offreedom of eye movement (e.g., azimuth and elevation), the fixed mirror14062 can be replaced with a second fovea tracking mirror (not shown)and one of the two fovea tracking mirrors can be used to track onedegree of freedom of eye movement and the second fovea tracking mirrorcan be used to track a second degree of freedom of eye movement. In suchan alternative, single degree of freedom fovea tracking mirrors may beused. Referring again to FIG. 36 , the third relay lens element 14060 incombination with the forth relay lens element 14066 forms a unitymagnification afocal relay. The fovea tracking mirror 14068 can add tothe deflection of the light beam 14028 produced by the image scanningmirror 14030 and thereby deflect the mean angle of the entire solidangle range of beam angles produced by the image scanning mirror 14030off axis in order to track the fovea (not shown) of a user's eye 210. Aneye-tracking camera 14098 tracks the eye gaze of a user's eye 210. Theeye-tracking camera 14098 is coupled to a fovea tracking control system14097. The eye-tracking camera 14098 outputs information indicative ofthe eye gaze which is input to the fovea tracking control system 14097.The fovea tracking control system 14097 is drivingly coupled to thefovea tracking mirror 14068. Based on the eye gaze information receivedfrom the eye-tracking camera 14098, the fovea tracking control system14097 outputs a signal to the fovea tracking mirror 14068 in order toorient the fovea tracking mirror 14068 to track the fovea of the user'seye 14099. The fovea tracking control system 14097 can use imageprocessing to determine the user's eye gaze and generate the signal tocontrol the fovea tracking mirror based on the eye gaze.

After being reflected by the fovea tracking mirror 14068 the lightpasses back through the fourth relay lens element 14066 and the thirdQWP 14064. The first pass of light through the third QWP 14064 convertsthe light to circularly polarized light, the reflection by the foveatracking mirror 14068 reverses the handeness of the circularly polarizedlight and the second pass through the third QWP 14064 converts the lightback to the S-polarized state. Because the light is now S-polarized itis reflected by the polarization selective reflector 14038 and deflectednominally 90 degrees toward the first QWP 14040. The first QWP 14040converts the S-Polarized light to left hand circularly polarized (LHCP)light. The light then passes through second relay lens element 14042.The fourth relay lens element 14066 in combination with the second relaylens element 14042 forms a unity magnification afocal compound lens. Therelay lens elements 14032, 14042, 14060, 14066 are symmetrically placedat 90 degree intervals about the center of the polarization selectivemirror 14038. Generally successive (in the order of light propagation)relay lens elements 14032, 14042, 14060, 14066 form unity magnificationafocal relays. Successive relay lens elements positioned so as to beconfocal, sharing a common focal point halfway across the PBS 14036. Therelay lens elements 14032, 14042, 14060, 14066 can include, by way ofnon-limiting examples, aspheric lenses, aplanatic lenses, hybridrefractive and diffractive lenses and achromatic lenses, compound lensesincluding for example refractive lenses along with diffractive lenses.As used in the present description “relay lens element” includes asingle lens or compound lens.

For LHCP light the first group geometric phase lens 14050 has a positiverefractive power which increases the refractive power of the first group14046. For LHCP the focal length of the first group 14044 is equal to adistance from the principal plane of the first group 14044 to a pointF_(LHCP). Upon passing through the first group geometric phase lens14050 the LHCP light is converted to RHCP light. Subsequently the lightpasses through the second group 14052. For RHCP light the second groupgeometric phase lens 14056 has a negative refractive power so that thepositive refractive power of the second group 14052 will be lower thanthe refractive power of the second group positive refractive lens 14054alone. For RHCP light the second group 14052 has a focal length equal toa distance from a principal plane of the second group 14052 to the pointF_(LHCP). Accordingly for LHCP light entering the dual magnificationafocal magnifier 14048, the dual magnification afocal magnifier 14048serves as a demagnifier with a magnification less than one. Thus a solidangle range of light beam directions produced by the image scanningmirror 14030 which is deflected by the fovea tracking mirror 14068 isdemagnified to cover a reduced angular range which tracks a user's foveaas the user's gaze is shifted. Recall that for incoming RHCP the dualmagnification afocal magnifier 14048 has a magnification greater thanone. The magnification greater than one is used to provide a wider fieldof view corresponding to a portion of the user's retina outside thefovea.

In certain embodiments the second group 14052 is a mirror image of thefirst group 14046, in which case the first group geometric phase lens14050 and the second group geometric phase lens 14056 are identical, andthe first group positive refractive lens 14044 and the second grouppositive refractive lens 14054 are identical. If the refractive lenses14044, 14054 have surfaces of different refractive power, they can bepositioned so that surfaces of the same refractive power face each otherin order to maintain the mirror image symmetry of the dual magnificationafocal magnifier 14048. In this case although each group 14046, 14052can have two different principal planes depending on whether thegeometric phase lenses 14050, 14056 are acting as positive or negativelenses, nonetheless two groups 14046, 14052 can be spaced from eachother at a fixed distance that maintains the confocal relation of thetwo groups 14046, 14052 in order to maintain the afocal magnification ofthe magnifier 14048 regardless of whether LHCP or RHCP light enteringthe magnifier 14048.

A set of three augmented reality glasses eyepiece waveguides including afirst eyepiece waveguide 14070, a second eyepiece waveguide 14072 and athird eyepiece waveguide 14074 are positioned beyond and opticallycoupled (through free space, as shown) to the second group 14052 of thedual magnification afocal magnifier 14048. Although three eyepiecewaveguides 14070, 14072, 14074 disposed in overlying relation are shown,alternatively a different number of eyepiece waveguides are provided.For example multiple sets of three eyepiece waveguides, with each setconfigured to impart a different wavefront curvature (corresponding to adifferent virtual image distance) to exiting light may be provided. Thethree eyepiece waveguides 14070, 14072, 14074 are respectively providedwith three light incoupling elements 14076, 14078, 14080 including afirst light incoupling element 14076, a second light incoupling element14078 and a third light incoupling element 14080. Each of the threeeyepiece waveguides 14070, 14072, 14074 can be configured to transferlight in a particular color channel, e.g., red, green or blue light.Additionally each of the incoupling elements 14076, 14078, 14080 can bewavelength selective so as to only couple light in one color channelinto its associated eyepiece waveguide 14070, 14072, 14074. Theincoupling elements 14076, 14078, 14080 can for example comprisespectrally selective reflective diffraction gratings, such as forexample diffraction gratings made of cholesteric liquid crystalmaterial. Such cholesteric liquid crystal material has a helical pitchwhich determines a spectral reflectivity band. Each of the incouplingelements can for example include two superposed layers of cholestericliquid crystal material with one being reflective of LHCP light and theother being reflective of RHCP light. Diffraction gratings generallyhave a profile pitch which determines light deflection angles. In thecase that the incoupling elements 14076, 14078, 14080 are implemented asdiffraction gratings the grating profile pitch of each grating issuitably selected in view of an associated the wavelength of light to beincoupled such that light is diffracted to angles above the criticalangle for total internal reflection for the associated eyepiecewaveguide 14070, 14072, 14074. The first, second and third eyepiecewaveguides 14070, 14072, 14074 respectively include a first exit pupilexpander (EPE) 14082, a second EPE 14084 and a third EPE 14086. The EPEs14082, 14084, 14086 may be implemented as transmissive and/or reflectivediffraction gratings. The EPEs 14082, 14084, 14086 incrementally couplelight that is propagating within the waveguides 14070, 14072, 14074 outof the waveguides 14070, 14072, 14074 such that light exits thewaveguides 14070, 14072, 14074 over a relatively wide area compared tothe transverse extent of the incoupling elements 14076, 14078, 14080.Orthogonal pupil expanders (OPEs) not visible in FIG. 36 can also beprovided on the eyepiece waveguides 14070, 14072, 14074 and locatedbehind the EPEs 14082, 14084, 14086. The OPEs serve to deflect lightfrom the incoupling elements 14076, 14078, 14080 that is propagatingwithin the eyepiece waveguides 14070, 14072, 14074 toward the EPEs14082, 14084, 14086. The OPEs may be located in the path of lightemanating from the incoupling elements 14076, 14078, 14080 and the EPEs14082, 14084, 14086 may be outside the path of light emanating from theincoupling elements 14076, 14078, 14080, but the OPEs may deflect lightfrom the incoupling elements 14076, 14078, 14080 toward the EPEs 14082,14084.

According to an alternative embodiment the first relay lens element14032 has a longer focal length than the second 14042, third 14060 andfourth 14066 relay lens elements, and is spaced from the center of thePBS 14036 (taking into account the index of refraction of the PBS 14036)by a distance equal to the longer focal length. In this case the longerfocal length first relay lens element 14032 in combination with thesecond relay lens 14042 imparts an angular magnification greater than1:1 to the non-fovea tracked light; and the longer focal length firstrelay lens element 14032 in combination with the third relay lenselement 14060 imparts an angular magnification greater than 1:1 to foveatracked light. Recall that the dual magnification afocal magnifier 14048will demagnify the fovea tracked light and the magnifiy the non-foveatracked light. Thus changing the focal length of the first relay lenselement 14032 provides another degree of design freedom that can be usedto set the magnifications achieved in the system 14000 withoutdisturbing the symmetry of the design of the dual magnification afocalmagnifier 14048. Introducing asymmetry into the design of the dualmagnification afocal magnifier 14048 is another possible alternative.

According to an alternative embodiment in lieu of the geometric phaselenses 14050, 14056 other types of dual state lenses are used. Accordingto one alternative actively driven electowetting liquid lenses may beused. According to another alternative lenses that include a liquidcrystal with its ordinary axis aligned in a specific direction overlyinga diffractive optic made of a material that matches the ordinary axisand exhibits a lens power for light polarized parallel to theextraordinary axis may be used. In the latter case the first QWP 14040may be eliminated as the anisotropic performance of the lenses will bedependent on the linear polarization differences between the foveatracked and non-fovea tracked light.

Each orientation of the image scanning mirror 14030 corresponds tocertain angular coordinates in the image space when the polarizationrotation switch 14034 is configured to transmit non-fovea-trackedP-polarized light. When the polarization rotation switch 14034 isconfigured to output S-polarized light that is fovea-tracked, theorientation of the image scanning mirror 14030 in combination with theorientation of the fovea tracking mirror 14068 determine angularcoordinates in the image space. The angles of light beam propagationdetermined by the orientation of the image scanning mirror and the foveatracking mirror 14068 are multiplied by the magnifications of the dualmagnification afocal magnifier 14048 and optionally by magnificationdetermined by the relative focal lengths of the relay lenses 14032,14042, 14060, 14066. The effective size of pixel defined in angularimage space is related to the inverse of the modulation rates of thelaser diodes 14002, 14010, 14016 and the angular rate of motion of theimage scanning mirror 14030. To the extent that the motion of the imagescanning mirror 14030 may be sinusoidal, the modulation rate of thelaser diodes 14002, 14010, 14016 may be made inversely related to theangular rate of the image scanning mirror 14030 in order to reduce oreliminate pixel size variation. When both fovea tracked and non-foveatracked are being generated the laser diodes 14002, 14010, 14016 thefull potential modulation rate of laser diodes 14002, 14010, 14010(limited by characteristics of available lasers) can be used (at leastfor certain points in the field of view), and the full angular range ofthe image scanning mirror can be used such that resolution imagery ofimagery produced for the fovea tracked region which subtends arelatively small solid angle range can be higher (smaller pixel size)than the resolution of imagery produced for the wider field of view.

According to certain embodiments in an augmented reality system in whichthe system 14000 is used virtual content is superimposed on the realworld which is visible to the user through the eyepiece waveguides14070, 14072, 14074. The virtual content is defined as 3D models (e.g.,of inanimate objects, people, animals, robots, etc.). The 3D models arepositioned and oriented in a 3D coordinate system. In an augmentedreality system, through the provision of, for example, an inertialmeasurement unit (IMU) and/or visual odometry the aforementioned 3Dcoordinate system is maintained registered to a real world environment(inertial reference frame) of the user of the augmented reality system.A game engine processes the 3D models taking into account their positionand orientation in order to render a left eye image and a right eyeimage of the 3D models, for output to the user via the system 14000 (anda like systems for the user's other eye). To the extent that the 3Dmodels are defined in a coordinate system that is fixed to user'senvironment and to the extent that the user may move and turn his or herhead (which carriers the augmented reality glasses) within theenvironment, the rendering of the left eye image and the right eye imageis updated to take into account the user's head movement and turning. Sofor example if a virtual book is displayed resting on a real table andthe user's rotates his or her head by 10 degrees to the left in responseto information of the rotation from the IMU or a visual odometrysubsystem (not shown), the game engine will update the left and rightimages to shift the image of the virtual book being output by the system14000 10 degrees to the right so that the book appears to maintain itsposition notwithstanding the user's head rotation. In the present caseimagery for a wider portion of the retina extending beyond the fovea andimagery for more limited portion of retina including the fovea are timemultiplexed through the system 14000 using the polarization rotationswitch 14034. Imagery is generated and output by the game engine insynchronism with the operation of the polarization rotation switch14034. As mentioned above the game engine generates left eye imagery andright eye imagery. The game engine also generates narrower FOV leftfovea and right fovea imagery which are output when the polarizationrotation switch 14034 is configured to output S-polarization light thatis fovea tracked using the fovea tracking mirror 14068. As discussedabove such fovea tracked imagery is converted to LHCP light and isdemagnified by the dual magnification afocal magnifier 14048. Suchdemagnification limits the angular extent to a narrow range includingthe fovea (or at least a portion thereof). The demagnification reducespixel size thereby increasing angular resolution for the fovea trackedimagery.

FIG. 37A is a schematic illustration of a dual magnification afocalmagnifier 14048 used in augmented reality near eye display system shownin FIG. 36 according to one embodiment.

FIG. 37B is a schematic illustration of a dual focal magnificationafocal magnifier 15000 that may be used in the augmented reality neareye display system 14000 shown in FIG. 36 in lieu of the afocalmagnifier 14048 according to other embodiments. The afocal magnifier15000 includes a lens group 15002 that includes a positive refractivelens 15004 and a first geometric phase lens 15006. The afocal magnifier15000 further includes a second geometric phase lens 15008 spaced at adistance from the first lens group 15002. The first geometric phase lens15006 and the second geometric phase lens 15008 have oppositehandedness. For light having a handedness matching the handedness of ageometric phase lens the geometric phase lens acts as a positive lensand for light having a handedness opposite to the handedness of thegeometric phase lens the geometric phase lens acts as a negative lens.Additionally upon propagating through a geometric phase lens thehandedness of the light is reversed. Accordingly when the firstgeometric phase lens 15006 is acting as a positive lens the secondgeometric phase lens 15008 will also be acting as a positive lens andwhen the first geometric phase lens 15006 is acting as a negative lensthe second geometric phase lens 15008 will also be acting as a negativelens. When the first geometric phase lens 15006 is acting as a negativelens the lens group 15002 will have a longer focal length than the focallength of the positive refractive lens 15004 alone. When the firstgeometric phase lens 15006 is acting as a positive lens the lens group15002 will have a shorter focal length than the focal length of thepositive refractive lens 15004 alone.

Recall that in the augmented reality near eye display system 14000 shownin FIG. 36 , the P-polarized light output by the polarization switch14034 passes directly through the PBS 14036, is not foveal tracked andis converted to RHCP light by the first QWP 14040; whereas S-polarizedlight output from the polarization rotation switch 14034 is routed so asto be reflected by the foveal tracking mirror 14068 and is eventuallyconverted to LHCP light.

The embodiment shown in FIG. 37B will be further described with theassumption that the first geometric phase lens 15006 is left handed andthe second geometric phase lens 15008 is right handed. It is furtherassumed, that as in the case of the system 14000 shown in FIG. 36 , LHCPlight is foveal tracked and RHCP is not foveal tracked light and carriesimagewise modulated light for a wider FOV (a wider portion of theretina). For LHCP light the first geometric phase lens 15006 acts as apositive lens and the lens group 15002 has a relatively short focallength corresponding to a distance from the lens group 15002 to a focalpoint F_(LHCP). In transmitting light the first geometric phase lens15006 converts the LHCP light to RHCP light for which the secondgeometric phase lens 15008 has a positive refractive power and a focallength equal to a distance from the second geometric phase lens 15008 tothe point F_(LHCP). In this case the afocal magnifier 15000 forms aKeplerian afocal magnifier. By proper selection (as will be describedfurther below) of the focal lengths of the positive refractive lens15004, the first geometric phase lens 15006 and the second geometricphase lens 15008, the magnification of the afocal magnifier 15000 in theKeplerian configuration can be chosen to be about 1:1 or another desiredvalue. Assuming for example that image scanning mirror 14030 has anoptical angular scan range of +/−10 degrees, such an angular range cansubstantially cover the fovea region of the retina.

For RHCP light entering the afocal magnifier 15000 the first geometricphase lens 15006 has a negative optical power and the lens group 15002has a relatively longer focal length corresponding to a distance fromthe lens group 15002 to a point F_(RHCP). The first geometric phase lens15006 converts the RHCP light to LHCP light for which the secondgeometric phase lens 15008 has a negative focal length corresponding toa distance from the second geometric phase lens 15008 to the pointF_(RHCP). In this case, the afocal magnifier 15000 is configured as aGalilean afocal magnifier and can have a magnification substantiallygreater than 1:1 for example 3:1. Thus the RHCP light entering theafocal magnifier (which is not fovea tracked) can provide imagewisemodulated light to a larger portion of the retina beyond the fovea(compared to the portion illuminated by the LHCP light. It should benoted that the systems 14000, 15000 can be reconfigured to reverse theroles the RHCP and LHCP light.

For a given focal length of the positive refractive lens 15004 and givenmagnitude of focal length of the first geometric phase lens 15004, thelens group 15002 will have one of two focal lengths equal to distancesfrom the lens group 15002 to the points F_(LHCP) and F_(RHCP), dependingon the handedness of incoming light (as described above). The secondgeometric phase lens 15008 should be positioned about half way betweenthe points F_(LHCP) and F_(RHCP) and the focal length of the secondgeometric phase lens 15008 should be set to about one-half of thedistance between F_(LHCP) and F_(RHCP). The magnification of theKeplerian configuration is equal to about minus the ratio of thedistance from the lens group 15002 to point F_(LHCP) divided by thedistance from the point F_(LHCP) to the second geometric phase lens15008. The magnification of the Galilean configuration is about equal tothe ratio of the distance from the lens group 15002 to the pointF_(RHCP) divided by the distance from the second geometric phase lens15008 to the point F_(RHCP).

The dual magnification afocal magnifiers 14048, 15000 can be used inother types of optical devices, including, by way of non-limitingexample, telescopes, binoculars, cameras and microscopes. In systems inwhich a real image is to be formed the afocal magnifiers 14048, 15000can be used in combination with additional optical elements (e.g.,lenses, convex mirrors).

Referring to FIG. 36 , according to an alternative embodiment, the fixedmirror 14062 is replaced with a second image scanning mirror, and asecond subsystem (like what is shown in FIG. 36 ) including laserdiodes, collimating lenses and RGB dichroic combining cube can be usedto provide RGB image modulated light to the second scanning mirror. Thesecond subsystem and second scanning mirror would be dedicated toproviding fovea-tracked light. In this case the polarization rotationswitch 14034 can be dispensed with and both fovea-tracked andnon-fovea-tracked light can be simultaneously produced. In such analternative all of the laser diodes would be oriented to injectP-polarized light into the PBS 14036.

IV. Tracking the Entire Field of View with Eye Gaze

According to some embodiments, instead of presenting the first imagestream at a static position as illustrated in FIGS. 26E-26F, both thefirst image stream and the second image stream can be dynamicallyshifted around according to the user's current fixation point. FIGS.38A-38B illustrates schematically an exemplary configuration of imagesthat can be presented to a user according to some embodiments. FIG. 38Ashows how the second image stream 16020 can be positioned substantiallyat the center of the first image stream 16010. In some embodiments, itmay be desirable to offset the second image stream 16020 from the centerof the first image stream. For example, since a user's field of viewextends farther in the temporal direction than the nasal direction itmay be desirable to have the second image stream 16020 offset towardsthe nasal side of the first image stream. During operation, the firstand second image stream can be persistently shifted in accordance withthe user's current fixation point as determined in real-time usingeye-gaze tracking techniques, as shown in FIG. 38B. That is, the firstimage stream 16010 and the second image stream 16020 can be shiftedaround in tandem such that the user is usually looking directly at thecenter of both image streams. It should be noted that the grid squaresin FIGS. 38A-38B represent schematically image points that, much likefields 3002, 3004 and 3006 as described above with reference to FIG. 24, are defined in two-dimensional angular space.

Similar to the embodiments depicted in FIGS. 26A-26B, the second imagestream 16020 represents a high-resolution image stream having arelatively narrow FOV that can be displayed within the boundaries of thefirst image stream 16010. In some embodiments, the second image stream16020 can represent one or more images of virtual content as would becaptured by a second, different virtual camera having an orientation inrender space that can be dynamically adjusted in real-time based on dataobtained using eye-gaze tracking techniques to angular positionscoinciding with the user's current fixation point. In these examples,the high-resolution second image stream 16020 can represent one or moreimages of virtual content as would be captured by a fovea-trackedvirtual camera such as the fovea-tracked virtual camera described abovewith reference to FIGS. 26A-26D. In other words, the perspective inrender space from which one or more images of virtual contentrepresented by the second image stream 16020 is captured can bereoriented as the user's eye gaze changes, such that the perspectiveassociated with the second image stream 5020E is persistently alignedwith the user's foveal vision.

For example, the second image stream 16020 can encompass virtual contentlocated within a first region of render space when the user's eye gazeis fixed at the first position as illustrated in FIG. 38A. As the user'seye gaze moves to a second position different from the first position,the perspective associated with the second image stream 16020 can beadjusted such that the second image stream 16020 can encompass virtualcontent located within a second region of render space, as illustratedin FIG. 38B. In some embodiments, the first image stream 16010 has awide FOV, but a low angular resolution as indicated by the coarse grid.The second image stream 16020 has a narrow FOV, but a high angularresolution as indicated by the fine grid.

FIGS. 39A-39B illustrate some of the principles described in FIGS.38A-38B using some exemplary images that can be presented to a useraccording to some embodiments. In some examples, one or more of theimages and/or image streams depicted in FIGS. 39A-39B may representtwo-dimensional images or portions thereof that are to be displayed at aparticular depth plane, such as one or more of the depth planesdescribed above with reference to FIG. 25B. That is, such images and/orimage streams may represent 3-D virtual content having been projectedonto at least one two-dimensional surface at a fixed distance away fromthe user. In such examples, it is to be understood that such imagesand/or image streams may be presented to the user as one or more lightfields with certain angular fields of view similar to those describedabove with reference to FIGS. 26A-26D and 28A-28B.

As depicted, the content of a first image stream 17010 includes aportion of a tree. During a first period of time represented by FIG.39A, eye-tracking sensors can determine a user's eye gaze (i.e., thefoveal vision) is focused at a first region 17010-1 within a viewableregion 17000. In this example, first region 17010-1 includes lowerbranches of the tree. A second image stream 17020 can be positionedwithin the first region 17010-1 and have a higher resolution than thefirst image stream. The first and second image streams can be displayedconcurrently or in rapid succession in a position determined tocorrespond to the user's current eye gaze.

During a second period of time represented by FIG. 39B, the user's eyegaze can be detected shifting to a second region 17010-2 within theviewable region 1500 that corresponds to upper branches of the tree. Asdepicted, during the second period of time, the position and content ofthe first and second image streams changes to correspond to the secondregion 17010-2. The content of both the first image stream 17010 andsecond image stream 17020 can include the second region 17010-2 of thetree. The first and second image streams can be displayed concurrentlyor in rapid succession. Further detected movements of the user's eyegaze can be accommodated in the same manner to keep both the first andsecond image streams aligned with the user's current eye gaze.

Similar to the embodiments illustrated in FIGS. 28C-28D, because thehigher resolution second image stream 17020 overlays the portion of thefirst image stream 17010 within the user's foveal vision, the lowerresolution of the first image stream 17010 may not be perceived ornoticed by the user. Furthermore, because the first image stream 17010having a wide field of view can encompass a substantial portion of theuser's vision, the user may be prevented from fully perceiving theboundaries of the light field display. Therefore, this technique canprovide an even more immersive experience to the user.

FIGS. 40A-40D illustrate schematically a display system 18000 forprojecting images to an eye of a user according to some embodiments. Thedisplay system 18000 includes an image source 18010. The image source18010 can be configured to project first light beam 18052 associatedwith a first image stream and second light beam 18054 associated with asecond image stream. The first image stream can be a wide FOV and lowresolution image stream, and the second image stream can be a narrow FOVand high resolution image stream, as discussed above with reference toFIGS. 38A-38B. In some embodiments, the first light beam 18052 and thesecond light beam 18054 can be time-division multiplexed,polarization-division multiplexed, wavelength-division multiplexed, orthe like.

The display system 18000 can further include a 2D scanning mirror 18020configured to reflect the first light beam 18052 and the second lightbeam 18054. In some embodiments, the 2D scanning mirror 18020 can betilted in two directions based on the fixation position of the user'seye, such that both the first light beam 18052 and the second light beam18054 can project the first image stream and the second image stream,respectively, at the user's foveal vision.

The display system 18000 can further include a switchable opticalelement 18040. Although the switchable optical element 18040 isillustrated as a single element, it can include a pair of sub switchableoptical elements that functions as a switchable relay lens assembly.Each sub switchable optical element can be switched to a first statesuch that it operates as an optical lens with a first optical power, asillustrated in FIGS. 40A and 40C, or be switched to a second state suchthat it operates as an optical lens with a second optical powerdifferent from the first optical power, as illustrated in FIGS. 40B and40D. Each sub switchable optical element can be, for example, a liquidcrystal varifocal lens, a tunable diffractive lens, a deformable lens,or a multifocal birefringent lens according to various embodiments.

In cases where the first light beam 18052 and the second light beam18054 are time-division multiplexed, the switchable optical element18040 and the scanning mirror 18020 can operate as follows. Assume thatthe user's eye gaze is fixed at a first position during a first timeperiod. The scanning mirror 18020 can be in a first orientation duringthe first time period so that the first light beam 18052 and the secondlight beam 18054 are directed toward a first position, as illustrated inFIGS. 40A and 40B. During a first time slot of the first time period(Stage A₁) when the image source 18010 outputs the first light beam18052, the switchable optical element 18040 can be switched to the firststate where it operates as an optical lens with the first optical poweras illustrated in FIG. 40A. During a second time slot of the first timeperiod (Stage A₂) when the image source 18010 outputs the second lightbeam 18054, the switchable optical element 18040 can be switched to thesecond state where it operates as an optical lens with the secondoptical power as illustrated in FIG. 40B. Thus, the first light beam18052 are angularly magnified more than the second light beam 18054, sothat the first light beam 18052 can present the first image stream witha wider FOV than that of the second image stream presented by the secondlight beam 18054.

Now assume that the user's eye gaze moves from the first position to asecond position during a second time period. The scanning mirror 18020can be in a second orientation during the second time period so that thefirst light beam 18052 and the second light beam 18054 are directedtoward a second position, as illustrated in FIGS. 40C and 40D. During afirst time slot of the second time period (Stage B₁) when the imagesource 18010 outputs the first light beam 18052, the switchable opticalelement 18040 can be switched to the first state where it operates as anoptical lens with the first optical power as illustrated in FIG. 40C.During a second time slot of the second time period (Stage B₂) when theimage source 18010 outputs the second light beam 18054, the switchableoptical element 18040 can be switched to a second state where itoperates as an optical lens with the second optical power as illustratedin FIG. 40D.

In cases where the first light beam 18052 and the second light beam18054 are polarization-division multiplexed, the switchable opticalelement 18040 can comprise a multifocal birefringent lens, so that itoperates as an optical lens with the first optical power for the firstlight beam 18052 as illustrated in FIGS. 40A and 40C, and operates as anoptical lens with the second optical power for the second light beam18054 as illustrated in FIGS. 40B and 40D.

In cases where the first light beam 18052 and the second light beam18054 are wavelength-division multiplexed, the switchable opticalelement 18040 can comprise a wavelength-dependent multifocal lens, sothat it operates as an optical lens with the first optical power for thefirst light beam 18052 as illustrated in FIGS. 40A and 40C, and operatesas an optical lens with the second optical power for the second lightbeam 18054 as illustrated in FIGS. 40B and 40D.

FIGS. 41A-41D illustrate schematically a display system 19000 forprojecting images to an eye of a user according to some otherembodiments. The display system 19000 can be similar to the displaysystem 18000, except that the switchable optical element 18040 can bedisposed on the surface of the scanning mirror 18020. For example, theswitchable optical element 18040 can be one or more substrates layeredon the surface of the scanning mirror 18020.

In some further embodiments, the switchable optical element 18040 can bepositioned elsewhere in the display system 19000. For example, it can bepositioned between the image source 18010 and the scanning mirror 18020.

In some other embodiments, a polarization beam splitter or a dichroicbeam splitter can be used to de-multiplex the first light beam 18052 andthe second light beam 18054 into two separate optical paths, but bothoptical paths intersect the reflective surface of the scanning mirror18020.

In other embodiments, more than two image streams can be presented tothe user so that the transition in resolution from the user's fixationpoint to the user's periphery vision is more gradual in appearance. Forexample, a third image stream having a medium FOV and medium resolutioncan be presented in addition to the first image stream and the secondimage stream. In such cases, additional relay lens assemblies and/orscanning mirrors can be utilized to provide additional optical paths forthe additional image streams.

Time Multiplexing Scheme

In some embodiments, the high-FOV low-resolution image stream (i.e., thefirst image stream) and the low-FOV high-resolution image stream (i.e.,the second image stream) can be time-division multiplexed.

FIG. 42 shows a graph illustrating an exemplary time-divisionmultiplexing pattern suitable for use with a high-FOV low-resolutionimage stream and a low-FOV high-resolution image stream. As illustrated,the high-FOV low-resolution image stream and the low-FOV high-resolutionimage stream are allocated at alternating time slots. For example, eachtime slot can be about one eighty-fifth of a second in duration. Thus,each of the high-FOV low-resolution image stream and the low-FOVhigh-resolution image stream may have a refresh rate of about 42.5 Hz.In some embodiments, an angular region corresponding to light fields ofthe low-FOV high-resolution image stream overlaps a portion of anangular region of the light fields corresponding to the high-FOVlow-resolution image stream making the effective refresh rate in theoverlapped angular region about 85 Hz (i.e., twice the refresh rate ofeach individual image stream).

In some other embodiments, the time slots for the high-FOVlow-resolution image stream and the time slots for the low-FOVhigh-resolution image stream can have different durations. For example,each time slot for the high-FOV low-resolution image stream can have aduration longer than one eighty-fifth seconds, and each time slot forthe low-FOV high-resolution image stream can have a duration shorterthan one eighty-fifth seconds, or vice versa.

FIG. 43 illustrates schematically a display system 21000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 21000 may share some elements in common with displaysystem 8000 as illustrated in FIGS. 30A-30B; for this reason, thedescription about those common elements in relation to FIGS. 30A-30B areapplicable here as well. An image source 21002 can be configured toprovide a high-FOV low-resolution image stream in a first polarizationstate and a low-FOV high-resolution image stream in a secondpolarization state contemporaneously. For example, the firstpolarization state can be a linear polarization in a first direction,and the second polarization state can be a linear polarization in asecond direction orthogonal to the first direction; or alternatively,the first polarization state can be a left-handed circular polarizationand the second polarization state can be a right-handed circularpolarization. Similar to the display system 8000 illustrated in FIGS.30A-30B, the display system 21000 includes a polarization beam splitter21004 for separating light beams projected by an image source (e.g.,image source 21002) into a first light beam associated with the high-FOVlow-resolution image stream propagating along a first optical path, anda second light beam associated with the low-FOV high-resolution imagestream propagating along a second optical path.

Similar to the display system illustrated in FIGS. 30A-30B, the displaysystem 21000 can include a first optical lens (lens A) positionedbetween the image source 21002 and the beam splitter 21004, a secondoptical lens (lens B) positioned downstream from the beam splitter 21004along the first optical path, and a third optical lens (lens C)positioned downstream from the beam splitter 21004 along the secondoptical path. In some embodiments, as described above in relation toFIGS. 30A-30B and 31A-31B, the combination of the first optical lens(lens A) and the second optical lens (lens B) can provide an angularmagnification for the first light beam that is greater than unity, andthe combination of the first optical lens (lens A) and the third opticallens (lens C) can provide an angular magnification for the second lightbeam that is substantially equal to unity or less than unity. Thus, thefirst light beam can project an image stream that has a wider FOV thanthat projected by the second light beam.

Similar to the display system 8000 illustrated in FIGS. 30A-30B, thedisplay system 21000 also includes a foveal tracker 21006 that can takethe form of a scanning mirror (e.g., a MEMs mirror), which can becontrolled based on the fixation position of the user's eye fordynamically projecting the second light beam associated with thelow-FOV, high-resolution image stream.

The display system 21000 can also include a first in-coupling grating(ICG) 21010 and a second ICG 21020 coupled to an eyepiece 21008. Theeyepiece 21008 can be a waveguide plate configured to propagate lighttherein. Each of the first ICG 21010 and the second ICG 21020 can be adiffractive optical element (DOE) configured to diffract a portion ofthe light incident thereon into the eyepiece 21008. The first ICG 21010can be positioned along the first optical path for coupling a portion ofthe first light beam associated with the high-FOV low-resolution imagestream into the eyepiece 21008. The second ICG 21020 can be positionedalong the second optical path for coupling a portion of the second lightbeam associated with the low-FOV high-resolution image stream into theeyepiece 21008.

The display system 21000 can also include a first switchable shutter21030, and a second switchable shutter 21040. The first switchableshutter 21030 is positioned along the first optical path between thesecond optical lens (lens B) and the first ICG 21010. The secondswitchable shutter 21040 is positioned along the second optical pathbetween the foveal tracker and the second ICG 21020. The operation ofthe first switchable shutter 21030 and the second switchable shutter21040 can be synchronized with each other such that the high-FOVlow-resolution image stream and the low-FOV high-resolution image streamare time-division multiplexed according to a time-division multiplexingsequence (e.g. as illustrated in FIG. 42 ). The first switchable shutter21030 can be open for a time period corresponding to a first time slotassociated with the high-FOV low-resolution image and closed during asecond time slot associated with the low-FOV high-resolution imagestream. Similarly, the second switchable shutter 21040 is open duringthe second time slot and is closed during the first time slot.

As such, the high-FOV low-resolution image stream is coupled into theeyepiece 21008 by way of the first ICG 21010 during the first time slot(e.g., when the first switchable shutter 21030 is open), and the low-FOVhigh-resolution image stream is coupled into the eyepiece 21008 by wayof the second ICG 21020 during the second time slot (e.g., when thesecond switchable shutter 21040 is open). Once the high-FOVlow-resolution image stream and the low-FOV high-resolution image streamare coupled into the eyepiece 21008, they may be guided and out-coupled(e.g., by out-coupling gratings) into a user's eye.

FIG. 44 illustrates schematically a display system 22000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 22000 may share some elements in common with the displaysystem 8000 illustrated in FIGS. 30A-30B; the description about thoseelements in relation to FIGS. 30A-30B are applicable here as well. Thehigh-FOV low-resolution image stream and the low-FOV high-resolutionimage stream provided by the image source 22002 can be time-divisionmultiplexed and can be in a given polarized state.

The display system 22000 can include a switchable polarization rotator22010 (e.g., ferroelectric liquid-crystal (FLC) cell with a retardationof half a wave). The operation of the switchable polarization rotator22010 can be electronically programed to be synchronized with the framerates of the high-FOV low-resolution image stream and the low-FOVhigh-resolution image stream in the time-division multiplexing (e.g., asillustrated in FIG. 42 ), so that the switchable polarization rotator22010 does not rotate (or rotates by a very small amount) thepolarization of the high-FOV low-resolution image stream, and rotatesthe polarization of the low-FOV high-resolution image stream by about 90degrees (i.e., introducing a phase shift of π), or vice versa.Therefore, after passing through the switchable polarization rotator22010, the polarization of the high-FOV low-resolution image stream maybe orthogonal to the polarization of the low-FOV high-resolution imagestream. For example, the high-FOV low-resolution image stream can bes-polarized, and the low-FOV high-resolution image stream can bep-polarized, or vice versa. In other embodiments, the high-FOVlow-resolution image stream can be left-handed circularly polarized, andthe low-FOV high-resolution image stream can be right-handed circularlypolarized, or vice versa.

The display system 22000 can include a polarization beam splitter 22004for separating light beams into a first light beam associated with thehigh-FOV low-resolution image stream propagating along a first opticalpath toward the first ICG 21010, and a second light beam associated withthe low-FOV high-resolution image stream propagating along a secondoptical path toward the second ICG 21020.

The display system 22000 can also include a static polarization rotator22020 positioned along one of the two optical paths, for example alongthe second optical path as illustrated in FIG. 44 . The staticpolarization rotator 22020 can be configured to rotate the polarizationof one of the low-FOV high-resolution image stream and the high-FOVlow-resolution image stream, so that the two image streams may havesubstantially the same polarization as they enter the first ICG 21010and the second ICG 21020, respectively. This may be advantageous incases where the first ICG 21010 and the second ICG 21020 are designed tohave a higher diffraction efficiency fora certain polarization. Thestatic polarization rotator 22020 can be, for example, a half-waveplate.

FIG. 45 illustrates schematically a display system 23000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 23000 may share some elements in common with the displaysystem 8000 illustrated in FIGS. 30A-30B; the description about thoseelements in relation to FIGS. 30A-30B are applicable here as well. Animage source 23002 can be configured to provide a high-FOVlow-resolution image stream and a low-FOV and high-resolution imagestream that are time-division multiplexed.

Here, instead of a beam splitter, the display system 23000 includes aswitchable reflector 23004. The switchable reflector 23004 can beswitched to a reflective mode where an incident light beam is reflected,and to a transmission mode where an incident light beam is transmitted.The switchable reflector may include an electro-active reflectorcomprising liquid crystal embedded in a substrate host medium such asglass or plastic. Liquid crystal that changes refractive index as afunction of an applied current may also be used. Alternatively, lithiumniobate may be utilized as an electro-active reflective material inplace of liquid crystal. The operation of the switchable reflector 23004can be electronically programed to be synchronized with the frame ratesof the high-FOV low-resolution image stream and the low-FOVhigh-resolution image stream in the time-division multiplexing (forexample as illustrated in FIG. 42 ), so that the switchable reflector23004 is in the reflective mode when the high-FOV low-resolution imagestream arrives, and in the transmission mode when the low-FOVhigh-resolution image stream arrives. Thus, the high-FOV low-resolutionimage stream can be reflected by the switchable reflector 23004 alongthe first optical path toward the first ICG 21010; and the low-FOVhigh-resolution image stream can be transmitted by the switchablereflector 23004 along the second optical path toward the second ICG21020.

Alternatively, the switchable reflector 23004 can be replaced by adichroic mirror configured to reflect light in a first set of wavelengthranges, and to transmit light in a second set of wavelength ranges. Theimage source 23002 can be configured to provide the high-FOVlow-resolution image stream in the first set of wavelength ranges, andthe low-FOV high-resolution image stream in the second set of wavelengthranges. For example, the first set of wavelength ranges can correspondto the red, green, and blue (RGB) colors, and the second set ofwavelength ranges can correspond to the RGB colors in a different huethan that of the first set of wavelength ranges. In some embodiments,the high-FOV low-resolution image stream and the low-FOV high-resolutionimage stream are time-division multiplexed, for example as illustratedin FIG. 42 . In some other embodiments, the high-FOV low-resolutionimage stream and the low-FOV high-resolution image stream are presentedsimultaneously.

Polarization Multiplexing Scheme

In some embodiments, the high-FOV low-resolution image stream and thelow-FOV high-resolution image stream can be polarization-divisionmultiplexed. An image source can include a first set of RGB lasers forproviding the high-FOV low-resolution image stream in a firstpolarization, and a second set of RGB lasers for providing the low-FOVhigh-resolution image stream in a second polarization different from thefirst polarization. For example, the high-FOV low-resolution imagestream can be s-polarized, and the low-FOV high-resolution image streamcan be p-polarized, or vice versa. Alternatively, the high-FOVlow-resolution image stream can be left-handed circular polarized, andthe low-FOV high-resolution image stream can be right-handed circularpolarized, or vice versa.

FIG. 46 illustrates schematically a display system 25000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 25000 may share some elements in common with the displaysystem 8000 illustrated in FIGS. 30A-30B; the description about thoseelements in relation to FIGS. 30A-30B are applicable here as well. Animage source 25002 can be configured to provide a high-FOVlow-resolution image stream and a low-FOV and high-resolution imagestream that are polarization-division multiplexed, as discussed above.

The display system 25000 can include a polarization beam splitter 25004for separating light beams into a first light beam associated with thehigh-FOV low-resolution image stream propagating along a first opticalpath toward the first ICG 21010, and a second light beam associated withthe low-FOV high-resolution image stream propagating along a secondoptical path toward the second ICG 21020.

The display system 25000 can also include a static polarization rotator25020 positioned along one of the two optical paths, for example alongthe second optical path as illustrated in FIG. 46 . The staticpolarization rotator 25020 can be configured to rotate the polarizationof one of the low-FOV high-resolution image stream and the high-FOVlow-resolution image stream, so that the two image streams may havesubstantially the same polarization as they enter the first ICG 21010and the second ICG 21020, respectively. This may be advantageous incases where the first ICG 21010 and the second ICG 21020 are designed tohave a higher diffraction efficiency fora certain polarization. Thestatic polarization rotator 25020 can be, for example, a half-waveplate.

V. Optical Architectures for Incoupling Images Projected into OpposingSides of the Eyepiece

In some embodiments, instead of having two ICGs laterally separated fromeach other (i.e., having separate pupils), a display system can beconfigured so that the high-FOV low-resolution image stream and thelow-FOV high-resolution image stream are incident on opposing sides ofthe same ICG (i.e., having a single pupil).

FIG. 47 illustrates schematically a display system 26000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 26000 can include a first image source 26002 configuredto provide a high-FOV low-resolution image stream, and a second imagesource 26004 configured to provide a low-FOV high-resolution imagestream.

The display system 26000 can also include a first optical lens (lens A)and a second optical lens (lens B) positioned along a first optical pathof the high-FOV low-resolution image stream. In some embodiments, thecombination of the first optical lens and the second optical lens canprovide an angular magnification that is greater than unity for a firstlight beam associated with the high-FOV low-resolution image stream,thereby resulting in a wider FOV for the first light beam.

The display system 26000 also includes an eyepiece 26008 and anin-coupling grating (ICG) 26010 coupled to the eyepiece 26008. Theeyepiece 26008 can be a waveguide plate configured to propagate lighttherein. The ICG 26010 can be a diffractive optical element configuredto diffract a portion of the light incident thereon into the eyepiece26008. As the first light beam associated with the high-FOVlow-resolution image stream is incident on a first surface 26010-1 ofthe ICG 26010, a portion of the first light beam is diffracted into theeyepiece 26008 in a reflection mode (e.g., a first order reflection),which may then be subsequently propagated through the eyepiece 26008 andbe out-coupled toward an eye of a user.

The display system 26000 can also include a third optical lens (lens C)and a fourth optical lens (lens D) positioned along a second opticalpath of the low-FOV high-resolution image stream. In some embodiments,the combination of the third optical lens and the fourth optical lenscan provide an angular magnification that is equal substantially tounity or less than unity for a second light beam associated with thelow-FOV high-resolution image stream. Thus, the second light beam mayhave a narrower FOV than that of the first light beam.

The display system 26000 can further include a foveal tracker 26006,such as a scanning mirror (e.g., a MEMs mirror), that can be controlledbased on the fixation position of the user's eye for dynamicallyprojecting the second light beam associated with the low-FOV andhigh-resolution image stream.

The second light beam associated with the low-FOV high-resolution imagestream may be incident on the second surface 26010-1 of the ICG 26010opposite the first surface 26010-2. A portion of the second light beamcan be diffracted into the eyepiece 2408 in a transmission mode (e.g., afirst order transmission), which may then be subsequently propagatedthrough the eyepiece 26008 and be out-coupled toward the eye of theuser.

As described above, the display system 26000 uses a single ICG 26010,instead of two separate ICGs as illustrated in FIGS. 43-46 . This cansimplify the design of the eyepiece.

FIG. 48 illustrates schematically a display system 27000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 27000 may share some elements in common with the displaysystem 8000 illustrated in FIGS. 30A-30B; the description about thoseelements in relation to FIGS. 30A-30B are applicable here as well. Thedisplay system 27000 can include an image source 27002 configured toprovide a high-FOV low-resolution image stream and a low-FOV andhigh-resolution image stream that are time-division multiplexed. In someembodiments, the image source 27002 can take the form of a picoprojector.

The display system 27000 can include a polarizer 27010 positioneddownstream from the image source 27002 and configured to convert thehigh-FOV low-resolution image stream and the low-FOV and high-resolutionimage stream from an unpolarized state into a polarized state, such asS-polarized and P-polarized, or RHCP and LHCP polarized.

The display system 27000 can further include a switchable polarizationrotator 27020 positioned downstream from the polarizer 27010. Theoperation of the switchable polarization rotator 27020 can beelectronically programed to be synchronized with the frame rates of thehigh-FOV low-resolution image stream and the low-FOV high-resolutionimage stream in the time-division multiplexing, so that the switchablepolarization rotator 27020 does not rotate (or rotates by a very smallamount) the polarization of the high-FOV low-resolution image stream,and rotates the polarization of the low-FOV high-resolution image streamby about 90 degrees (i.e., introducing a phase shift of π), or viceversa. Therefore, after passing through the switchable polarizationrotator 27020, the polarization of the high-FOV low-resolution imagestream may be orthogonal to the polarization of the low-FOVhigh-resolution image stream. For example, the high-FOV low-resolutionimage stream can be s-polarized, and the low-FOV high-resolution imagestream can be p-polarized, or vice versa. In other embodiments, thehigh-FOV low-resolution image stream can be left-handed circularpolarized, and the low-FOV high-resolution image stream can be aright-handed circular polarized, or vice versa.

The display system 27000 further includes a polarization beam splitter27004 configured to reflect the high-FOV low-resolution image streamalong a first optical path, and to transmit the low-FOV high-resolutionimage stream along a second optical path.

The display system 27000 can further include a first optical lens (lensA) positioned in in front of the polarization beam splitter 27004, asecond optical lens (lens B) positioned downstream from the polarizationbeam splitter 27004 along the first optical path, and a third opticallens (lens C) positioned downstream from the beam splitter 27004 alongthe second optical path. In some embodiments, as described above inrelation to FIGS. 30A-30B and 31A-31C, the combination of the firstoptical lens (lens A) and the second optical lens (lens B) can providean angular magnification for the high-FOV low-resolution image streamthat is greater than unity; and the combination of the first opticallens (lens A) and the third optical lens (lens C) can provide an angularmagnification for the low-FOV high-resolution image stream that equalssubstantially to unity or less than unity. Thus, the high-FOVlow-resolution image stream may be projected to an eye of a user with awider FOV than that projected by the low-FOV high-resolution imagestream.

The display system 27000 can further include a foveal tracker 27006,such as a scanning mirror (e.g., a MEMs mirror), that can be controlledbased on the fixation position of the user's eye for dynamicallyprojecting the second light beam associated with the low-FOV andhigh-resolution image stream.

The display system 27000 can further include an eyepiece 27008 and anin-coupling grating (ICG) 27050 coupled to the eyepiece 27008. Theeyepiece 27008 can be a waveguide plate configured to propagate lighttherein. The ICG 27050 can be a diffractive optical element configuredto diffract a portion of the light incident thereon into the eyepiece27008.

The display system 27000 can further include a first reflector 27030positioned downstream from the second optical lens (lens B) along thefirst optical path. The first reflector 27030 can be configured toreflect the high-FOV low-resolution image stream toward the ICG 27050.As a first light beam associated with the high-FOV low-resolution imagestream is incident on a first surface 27050-1 of the ICG 27050, aportion of the first light beam is diffracted into the eyepiece 27008 ina transmission mode (e.g., a first order transmission), which maysubsequently propagate through the eyepiece 27008 and be out-coupledtoward an eye of a user.

The display system 27000 can further include a second reflector 27040positioned downstream from the foveal tracker 27006 along the secondoptical path. The second reflector 27040 can be configured to reflectthe low-FOV high-resolution image stream toward the ICG 27050. As asecond light beam associated with the low-FOV high-resolution imagestream is incident on a second surface 27050-2 of the ICG 27050 oppositeto the first surface 27050-1, a portion of the second light beam isdiffracted into the eyepiece 27008 in a reflective mode (e.g., a firstorder reflection), which may subsequently propagate through the eyepiece27008 and be out-coupled toward the eye of the user.

FIG. 49 illustrates schematically a display system 28000 for projectingimage streams to an eye of a user according to some embodiments. Thedisplay system 28000 is similar to the display system 27000, except thatit does not include an ICG. Instead, the display system 28000 includes afirst in-coupling prism 28030 (in place of the first reflector 27030 inthe display system 27000) for coupling the high-FOV low-resolution imagestream into the eyepiece 27008, and a second in-coupling prism 28040 (inplace of the second reflector 27040 in the display system 27000) forcoupling the low-FOV high-resolution image stream into the eyepiece27008. The index of refraction of the first in-coupling prism 28030 andthe index of refraction of the second in-coupling prism 28040 can besuitably selected with respect to the index of refraction of theeyepiece 27008, so that a fraction of the power contained in a firstlight beam associated with the high-FOV low-resolution image stream anda fraction of the power contained in a second light beam associated withthe low-FOV high-resolution image stream are coupled into the eyepiece27008 by the first in-coupling prism 28030 and the second in-couplingprism 28040, respectively.

VI. High Field of View and High Resolution Foveated Display usingOverlapping Optical Paths

In some embodiments, a display system may be configured so that thehigh-FOV low-resolution image stream and the low-FOV high-resolutionimage stream are provided to an eyepiece without utilizing a PBS toseparate a composite image stream into two image streams that propagatein different directions. Rather, the high-FOV low-resolution imagestream and the low-FOV high-resolution image stream may takesubstantially the same path from an image source to the eyepiece, whichmay obviate the PBS. This may have advantages for providing a compactform factor for the display system.

FIG. 50 illustrates schematically a display system 50000 for projectingimage streams to an eye of a user. The display system 50000 may includean image source 50002 (sometimes referred to as a light source)configured to provide a high-FOV low-resolution image and also provide alow-FOV high-resolution image (using, e.g., a high-FOV low-pixel-densityimage stream and a low-FOV high-pixel-density image stream,respectively). In some embodiments, the image source 50002 may providethe high-FOV low-resolution image stream and the low-FOV high-resolutionimage stream in a time-multiplexed manner, such as by interleavingframes from the high-FOV low-resolution image stream with frames of thelow-FOV high-resolution image stream.

The display system 50000 may also include variable optics 50004. In someembodiments, the variable optics 50004 may provide a different angularmagnification for light rays 50030 associated with the high-FOVlow-resolution image stream than for light rays 50020 associated withthe low-FOV high-resolution image stream, thereby enabling projection ofthe high-FOV low-resolution image stream out of the waveguide 50010 toprovide a wider FOV than that projected by the low-FOV high-resolutionimage stream. It will be appreciated that the range of angles at whichin-coupled light is incident on the ICG 50006 is preferably preservedupon the out-coupling of that light from the waveguide 50010. Thus,in-coupled light incident on the ICG 50006 at a wide range of anglesalso propagates away from the waveguide 50010 at a wide range of anglesupon being out-coupled, thereby providing a high FOV and more angularmagnification. Conversely, light incident on the ICG 50006 at acomparatively narrow range of angles also propagates away from thewaveguide 50010 at a narrow range of angles upon being out-coupled,thereby providing a low FOV and low angular magnification.

Additionally, to select the appropriate level of angular magnification,variable optics 50004 may alter light associated with the high-FOVlow-resolution image stream so that it has a different optical propertythen light associated with the low-FOV high-resolution image stream.Preferably, the function of the variable optics 50004 and the propertiesof light of each image stream are matched such that changing therelevant property of the light changes the optical power and focallength provided by the variable optics 50004. For example, the high-FOVlow-resolution image stream may have a first polarization and thelow-FOV low-resolution image stream may have a second polarization.Preferably, the variable optics 50004 is configured to provide differentoptical power and different focal lengths for different polarizations oflight propagating through it, such that the desired optical power may beselected by providing light of a particular, associated polarization.The first polarization may be a right hand circular polarization (RHCP),a left hand circular polarization (LFCP), S-polarization,P-polarization, another polarization type, or un-polarized. The secondpolarization may be a right hand circular polarization (RHCP), a lefthand circular polarization (LFCP), S-polarization, P-polarization,another polarization type, or un-polarized, so long as it is differentfrom the first polarization. In some preferred embodiments, the firstpolarization is one of a right hand circular polarization (RHCP) and aleft hand circular polarization (LFCP), and the second polarization isthe other of the left hand circular polarization (LFCP) and right handcircular polarization (RHCP).

In some embodiments, the operation of the variable optics 50004 may beelectronically programed to be synchronized with the frame rates of thehigh-FOV low-resolution image stream and the low-FOV high-resolutionimage stream in the time-division multiplexing. In some embodiments, theimage frames of the high-FOV stream are given their desired polarizationand angular magnification to couple to waveguide 50010 via ICG 50006while interleaved frames of the low-FOV stream are given their desiredmagnification and polarization to initially pass through ICG 50006, bepassed to mirror 50008, be targeted to the user's fixation point, andthen be coupled to waveguide 50010 via ICG 50006.

The display system 50000 also includes an eyepiece 50010 and apolarization-sensitive in-coupling grating (ICG) 50006 coupled to theeyepiece 50010. The eyepiece 50010 may be a waveguide, e.g., a plateplate, configured to propagate light therein, e.g., by total internalreflection. The polarization-sensitive ICG 50006 may be apolarization-sensitive diffractive optical element configured todiffract a portion of the light incident thereon into the eyepiece50010. In some embodiments, the ICG 50006 may be polarization-sensitivein that incident light having a particular polarization ispreferentially diffracted into the eyepiece 50010, while incident lightof at least one other polarization passes through the ICG 50006. Lightthat passes through the ICG 50006 without coupling into the eye piece50010 may be directed towards mirror 50008, which may be a MEMS mirror,and which may be configured to switch the polarization of incidentlight. As a first example, the polarization-sensitive ICG 50006 maycouple light having a right-hand circular polarization (RHCP) into thewaveguide, while passing light having a left-hand circular polarization(LHCP) through towards mirror 50008. As a second example,polarization-sensitive ICG 50006 may couple light having a LHCP into thewaveguide, while passing light having a RHCP through towards mirror50008.

In at least some embodiments, light reflected off of mirror 50008 may bedirected towards ICG 50006. Additionally, the reflection of the lightoff mirror 50008 may alter the polarization of the light (e.g., flip thepolarization of the light from RHCP to LHCP and vice versa) such thatthe reflected light has the desired polarization to be diffracted by ICG50006 and coupled into eye piece 50010. As an example, if ICG 50006 isconfigured to couple light having a RHCP into eye piece 50010, thenlight associated with the high FOV stream may be given a RHCP byvariable optics 50004 and then coupled into eye piece 50010. In such anexample, light associated with the low FOV stream may be given a LHCP byvariable optics 50004, such that the LHCP light may then pass throughICG 50006 without coupling into eyepiece 50001 and instead may bedirected towards mirror 50008. Reflection of the LHCP light off of themirror 50008 may flip the polarization of the light to RHCP. Then, whenthe now-RHCP light hits ICG 50006, it may be coupled by ICG 50006 intoeye piece 50010. Similar examples apply when ICG 50006 is configured tocouple LHCP into eye piece 50010.

As disclosed herein, mirror 50008 may be a movable mirror, e.g., ascanning mirror, and may function as a fovea tracker. As also discussedherein, the mirror 50008 may be controlled and moved/tilted based on thedetermined fixation position of the user's eye. The tilting of themirror 50008 may cause the reflected light to in-couple into thewaveguide 500010 at different locations, thereby causing light to alsoout-couple at different locations corresponding to the location of thefovea of the user's eye.

With continued reference to FIG. 50 , the light source 50002 may producea high-FOV low-resolution (HFLR) image stream and a low-FOVhigh-resolution (LFHR) image stream in a time-multiplexed manner.Additionally, the variable optics 50004 may alter the HFLR image streamto have a particular polarization (such as RHCP) (and associated angularmagnification) so that the HFLR image stream is coupled into waveguide50010 by polarization-sensitive ICG 50006. The variable optics may alterthe LFHR image stream to have a different polarization (such as LHCP)and associated angular magnification. As a result, the LFHR image streampasses through polarization-sensitive ICG 50006, reflects off of mirror50008 (flipping the polarization to RHCP and targeting the LFHR imagesto a user's fixation position), and is then coupled into waveguide 50010by ICG 50006.

FIG. 51 illustrates an example of an implementation of variable optics50004. As shown in FIG. 51 , variable optics 50004 may be formed frompolarizer 50012, switchable quarter wave plate (QWP) 50013, lens 50014,diffractive waveplate lens 50015, diffractive waveplate lens 50016, andlens 500017. This is merely one possible implementation of variableoptics 50004.

The polarizer 50012 may be configured to convert the high-FOVlow-resolution image stream and the low-FOV high-resolution image streamfrom light source 50002 from an unpolarized state into a polarizedstate, such as S-polarized and P-polarized, or RHCP and LHCP polarized.

The switchable QWP 50013 may be configured to convert the polarizedlight from polarizer 50012 into either (1) a right-hand circularpolarization (RHCP) or (2) a left-hand circular polarization (LHCP).

After exiting the QWP 50013, the light may be incident on lens 50014 anddiffractive waveplate lens 50015. The diffractive waveplate lens 50015may be a geometric phase lens including patternwise aligned liquidcrystal material. Diffractive waveplate lens 50015 may have a positiveoptical power (e.g., be a positive lens) for circularly polarized lightthat has a handedness (RH or LH) that matches their handedness and mayhave a negative optical power (e.g., be a negative lens) for circularlypolarized light of opposite handedness. Diffractive waveplate lens 50015may also have the property that it reverses the handedness of circularlypolarized light. Thus, if diffractive waveplate lens 50015 isright-handed and receives RHCP light from lens 500014, the diffractivewaveplate lens 50015 would act as a positive lens and the light would beleft-handed after passing through diffractive waveplate lens 50015.

After exiting the diffractive waveplate lens 50015, the light will beincident on diffractive waveplate lens 50016 and then lens 50017.Diffractive waveplate lens 50016 may operate in a manner similar to thatof diffractive waveplate lens 50015. Additionally, the handedness ofdiffractive waveplate lens 50016 may match that of diffractive waveplatelens 50015, at least in some embodiments. With such an arrangement, theoptical power of the diffractive waveplate lens 50016 will be oppositethat of diffractive waveplate lens 50015. Thus, in an example in whichthe switchable QWP 50013 provides light with a polarization matchingdiffractive waveplate lens 50015, lens 50015 will have a positiveoptical power and will also reverse the handedness of the light. Then,when the subsequent diffractive waveplate lens 50016 receives the light,lens 50015 will have a negative optical power, as it receives the lightafter its handedness was reversed.

With an arrangement of the type shown in FIG. 51 , the variable optics50004 may provide a first angular magnification when the switchable QWP50013 provides light matching the handedness of diffractive waveplatelens 50015 (e.g., such that lens 50015 provides a positive opticalpower, while lens 50016 provides a negative optical power) and mayprovide a second angular magnification when the switchable QWP 50013provides light of opposite handedness (e.g., such that lens 50015provides a negative optical power, while lens 50016 provides a positiveoptical power). In other embodiments, the handedness of the twodiffractive waveplate lens 50015 and 50016 may be different.

With reference now to FIGS. 52A-52B, additional details regardingexample ICG configurations are provided. For example, it will beappreciated that polarization sensitive ICG's may preferentially directlight in a particular lateral direction depending upon which side of theICG the light is incident. For example, with reference to FIG. 52A,light incident on ICG 50006 from below is redirected to the left of thepage. However, light incident on ICG 50006 from above would beundesirably directed towards the right of the page, away from the areaof the waveguide from which light is out coupled to a viewer. In someembodiments, in order to in-couple light such that it propagates in thedesired direction, different ICG's may be used for light incident fromdifferent directions or sides of the waveguide 50010.

For example, in some embodiments, the display system may be configuredso that the high-FOV low-resolution image stream and the low-FOVhigh-resolution image stream are coupled into waveguide 50010 (which maybe an eyepiece) using a pair of a polarization-sensitive in-couplinggratings (ICG) 50006 and 50040. Such an arrangement may be beneficialwhere, e.g., light that strikes an ICG from below (in the perspective ofFIGS. 50-53B) is coupled into the waveguide 50010 in a desired lateraldirection (to the left), while light that strikes the ICG from above iscoupled into the waveguide 50010 in the opposite direction (to theright). More details about in-coupling gratings (ICG) gratings aredescribed in U.S. patent application Ser. No. 15/902,927, the contentsof which are hereby expressly and fully incorporated by reference intheir entirety, as though set forth in full.

FIGS. 52A-52B illustrate schematically a display system 52000 forprojecting image streams to an eye of a user according to someembodiments of the present invention, which may include two ICGs 50006and 50040. In some embodiments, ICGs 50006 and 50040 may both beconfigured to couple light of the same polarization-type into waveguide50010. As an example, ICGs 50006 and 50040 may each couple light havinga left-hand circular polarization (LHCP) into waveguide 50010, whilepassing light having a right-hand circular polarization (RHCP).Alternatively, the polarizations may be swapped.

As shown in FIG. 52A, optical elements such as those shown in FIGS.50-51 may provide a high FOV low resolution image stream 50030 having aleft-handed circular polarization (LHCP). The light 50030 may beincident upon ICG 50006. Since the light 50030 is LHCP and the ICG 50006is configured to couple LHCP light into waveguide 50010, the light iscoupled by ICG 50006 into the waveguide 50010.

As shown in FIG. 52B, optical elements such as those shown in FIGS.50-51 may provide a low FOV high resolution image stream 50020 (whichmay be interleaved with the image stream of FIG. 52A in atime-multiplexed manner) having a right-handed circular polarization(RHCP). The light 50020 may be incident upon ICG 50006. However, sincethe light 50020 is RHCP and the ICG 50006 is configured to couple onlyLHCP light into waveguide 50010, the light 50020 passes through ICG50006. ICG 50040 may, similarly, be configured to couple only LHCP lightinto waveguide 50010, thus the light may also pass through ICG 50040.After passing through both ICGs, the light 50020 may be incident onmovable mirror 50008, which may be in a particular orientation basedupon a user's fixation point (as discussed herein in various sections).After reflecting off of mirror 50008, the polarization of the light50020 may be flipped, so the light is now LHCP. Then, the light 50020may be incident on ICG 50040, which may couple the now-LHCP light 50020into the waveguide 50010.

In some embodiments, the display system may be configured so that thehigh-FOV low-resolution image stream and the low-FOV high-resolutionimage stream are formed by light having the same polarization. As aresult, both image streams may be in coupled by the same ICG, upon beingincident on the same side of that ICG.

FIGS. 53A-53B illustrate schematically a display system 53000 forprojecting image streams to an eye of a user according to someembodiments of the present invention, which may include a single ICG50006 and a switchable reflector 50042. The switchable reflector 50042may be a liquid-crystal based planar device that switches between asubstantially transparent state and a substantially reflective state ata sufficiently high rate; that is, the switching rate of the switchablereflector 50042 is preferably sufficiently high to allow coordinationwith interleaved frames of the high-FOV low-resolution image stream andthe low-FOV high-resolution image stream. For example, the switchablereflector 50042 is preferably able to switch between reflective andtransmissive states at at least the same rate as the high and low-FOVresolution image streams are switched.

As shown in FIG. 53A, the ICG 50006 may receive a high FOV lowresolution image stream 50030 from optical elements such as those shownin FIGS. 50-51 . As an example, the image stream may have a left-handedcircular polarization (LHCP). The light of the image stream 50030 may beincident upon ICG 50006. However, ICG 50006 may be configured to coupleRHCP light and pass LHCP light. Thus, the LHCP light 50030 may passthrough ICG 50006. The light may then be incident on switchablereflector 50042, which may be configured in its reflective state (whilethe system is projecting high FOV low resolution image stream 50030).Thus, the light of the image stream 50030 may reflect off of switchablereflector 50042, thereby reversing the handedness of its polarization.After reflecting off of switchable reflector 50042, the 50030 light maybe incident again upon ICG 50006, and ICG 50006 may couple the now-RHCPlight 50030 into the waveguide 50010.

As shown in FIG. 53B, optical elements such as those shown in FIGS.50-51 may provide a low FOV high resolution image stream 50020 having aleft-handed circular polarization (LHCP). This arrangement differsslightly, in that the polarization of the low FOV image stream 50020matches the polarization of the high FOV image stream 50030. Such anarrangement may be achieved using a modification of the variable optics50004 shown in FIGS. 50-51 . As an example, an additional polarizer,e.g., a switchable polarizer, and may be provided between lens 50017 andICG 50006.

Returning to the low FOV high-resolution LHCP light 50020 in FIG. 53B,the light 50020 is incident upon ICG 50006. However, ICG 50006 isconfigured to couple RHCP into waveguide 50010. Thus, the light 50020passes through ICG 50006. The light 50020 is next incident upon theswitchable reflector 50042, which may be configured to be in itstransparent state (while the system is projecting low FOV highresolution light 50020). Thus the light may pass through switchablereflector 50042 and be incident upon mirror 50008 and, optionally, betargeted by mirror 50008 on a user's fixation point (as discussed hereinin various sections). After reflecting off of mirror 50008, thepolarization of the light 50020 may be flipped, so the light is nowRHCP. Then, the light 50020 may be incident on ICG 50006, which maycouple the now-RHCP light 50020 into the waveguide 50010. It will beappreciated that the mirror 50008 may be configured to provide foveatracking and/or may be sufficiently spaced from the ICG 50006 to accountfor the different focal length of the wearable optics 50004 (FIGS. 50-51), to provide a focused image.

Three-Dimensional Foveated Rendering

As described herein, a wearable display system (e.g., the wearabledisplay system 60) may present augmented or virtual reality content to auser. In an effort to reduce the processing power required to presentthe content (e.g., virtual content as described herein), and thusadditionally to reduce power requirements, FIGS. 10A-23 and the relateddiscussion describe adjusting various display characteristics of virtualcontent based on their respective three-dimensional locations within auser's field of view. For example, virtual content presented proximateto a three-dimensional location at which a user is fixating may bepresented (e.g., rendered) at a high resolution. As another example,resolution of virtual content may be reduced based on athree-dimensional distance of the virtual content from the user'sfixation point. By tying reductions in resolution to proximity to theuser's three-dimensional fixation point, the system may advantageouslylimit an extent to which such reductions in resolution are perceptible,as discussed herein.

Resolution may encompass any modification to a virtual object to alter aquality of presentation of the virtual object. Such modifications mayinclude one or more of adjusting a polygon count of the virtual object,adjusting primitives utilized to generate the virtual object (e.g.,adjusting a shape of the primitives, for example adjusting primitivesfrom triangle mesh to quadrilateral mesh, and so on), adjustingoperations performed on the virtual object (e.g., shader operations),adjusting texture information, adjusting color resolution or depth,adjusting a number of rendering cycles or a frame rate, and so on,including adjusting quality at one or more points within a graphicspipeline of graphics processing units (GPUs). Additionally, in someembodiments, virtual content located proximate to a user's fixationpoint may be presented at a higher refresh rate than virtual contentfarther from the fixation point.

As described above, FIG. 10A illustrates a virtual object 1008A beinglocated proximate to a user's fixation point (e.g., a three-dimensionalvergence point 1006). In the example of FIG. 10A, the virtual object1008A is presented (e.g., rendered) in high-resolution in a renderedframe 1010 to the user. In contrast, virtual object 1008B, which islocated farther from the fixation point, is rendered in low resolutionin the rendered frame 1010. To identify a resolution at which to rendervirtual content, the wearable display system may separate the user'sfield of view into different resolution adjustment zones. For example,FIG. 11A1 illustrates example resolution adjustment zones (also referredto herein as ‘zones’) encompassing different three-dimensional volumesof space within a user's field of view. As illustrated, each resolutionadjustment zone may be assigned a particular resolution. In the exampleof FIG. 11A1, an assigned resolution represents a polygon countassociated with rendering virtual content located within a resolutionadjustment zone. FIGS. 11A2-11E illustrate some additional examplesschemes to separate a user's three-dimensional field of view intoresolution adjustment zones.

As described above, these resolution adjustment zones may be customizedaccording to user settings. For example, a user may update a size,shape, location, and so on, of resolution adjustment zones.Additionally, applications or content may update the settings of theseresolution adjustment zones. As an example, a first application mayprefer that resolution sharply falls off based on distance to the user'sfixation point. The first application may implement stark depth of fieldadjustment through blurring, applying bokeh, and so on, to virtualcontent located away from the user's fixation point. Therefore, thefirst application may provide cinematic effects to virtual content beingpresented to the user.

To ensure that adjustment of resolution has low perceptibility (e.g., issubstantially imperceptible), the wearable display system may utilizeempirically determined schemes to identify resolutions at which torender differently located virtual content. For example, a userutilizing the wearable display system may train the system based on theuser's own visual perception. The wearable display system may presentdifferent types of virtual content to the user. The wearable displaysystem may also customize presentation of these types based on responsesreceived from the user. As another example, the wearable display systemmay utilize aggregated information from multitudes of users to identifya standard scheme to adjust resolution. FIGS. 54-59 describe techniquesto determine shapes, sizes, and so on, of the resolution adjustmentzones illustrated in FIGS. 11A1-11E. While the description below focuseson identifying an angular distance encompassed by a resolutionadjustment zone, it should be understood that the description below mayat least be applied to any of the zones described in FIGS. 11A1-11E.

FIG. 54 illustrates a representation 5402 of an angular field of view ofa user along with an example resolution distribution 5410. In thisillustration, a user's field of view is separated according to angulardistance from a center 5404 of the field of view. The center 5404 maycorrespond to a foveal region of the user's field of view, which in theexample of FIG. 54 spans five degrees of the field of view. Thus, a usermay be able to resolve and identify fine detail of virtual objects thatfall within the center 5404. Other portions of the field of view (e.g.,portion 5408) are located further from the center 5404 and a user mayhave reduced visual acuity with respect to virtual content in theseportions. For example, a user may be unable to resolve fine details invirtual content presented in portion 5408. In this example, the user mayrotate or adjust his/her eyes to fixate at this virtual content and thusshift center 5404. Thus, the virtual content may be presented closer to,or within, the center 5404.

The representation 5402 of the user's field of view 5402 mayadditionally encompass an entire three-dimensional volume of real-worldspace visible to the user; that is, the representation 5402 may be aslice of the user's field of view (e.g., along a z axis). Thus, therepresentation 5402 may extend along two axes (e.g., an X and Z axis).It should be understood that the representation 5402 may be extendedalong a remaining third axis (e.g., a Y axis) and the techniquesdescribed herein may still be utilized. For example, a zone referred toherein as a foveal zone 5406 (e.g., a zone for which virtual content ispresented on a user's fovea) may be extended along the third axis.

Without being constrained by theory, a user may also have sufficientvisual acuity to identify details of virtual content that falls outsideof the foveal zone 5406. For example, the foveal zone 5406 maycorrespond to a fovea, but a high resolution zone 5418 may encompassvirtual objects that fall within a threshold angular distance from thefoveal zone 5406. The high resolution zone 5418, for example, mayinclude the fovea, parafovea belt, parafovea outer region, and so on. Asillustrated, the high resolution zone 5418 is represented as beingeighteen degrees. Thus, reducing resolution of virtual content presentedwithin this high resolution zone 5418 may be perceptible. The highresolution zone 5418 may therefore represent a high resolution region ortunnel for which virtual content is rendered at greater than a thresholdresolution (e.g., a lowest resolution 5416). As will be described,virtual content presented within the high resolution zone 5418 may berendered at reduced resolutions according to a resolution distribution5410.

The example resolution distribution 5410 of FIG. 54 identifies thefoveal zone 5406 as being a plateau of the high resolution zone 5418.For example, the distribution 5410 may be a Gaussian distribution, asuper Gaussian distribution, a normal or “bell curve” distribution, aCauchy distribution, and so on. In some embodiments, the distribution5410 may be governed by a function of mathematical equivalence orresemblance to a function describing the frequency response of one ofany of a variety of different types of filters, including linearbandpass filters, pulse shaping filters, and other types of signalprocessing filters. For example, in at least some of these embodiments,the distribution 5410 may be governed by a mathematical functionequivalent or similar to that which describes the frequency response ofa raised-cosine filter, a root-raised-cosine filter, a sinc filter, aGaussian filter, a Butterworth filter, a Chebyshev filter, a Besselfilter, and so on. As illustrated, the distribution 5410 identifies aresolution at which to render virtual content based on the virtualcontent's angular distance from a center of a field of view. While thedistribution 5410 is illustrated as being dependent on angular distance,it should be understood that the distribution 5410 may further bedependent on depth (e.g., different depths along a same angular distancemay be rendered at different resolutions). For example, and as will bedescribed in FIG. 56 , the resolution distribution 5410 may be amultivariate normal distribution.

A wearable display may utilize, at least in part, the distribution 5410to render virtual content. For example, virtual content within thefoveal zone 5406 may be rendered at a maximum resolution. However,virtual content presented outside of this foveal zone 5406 may berendered at decreasing resolutions. FIG. 54 illustrates that for anyangular distance, a particular resolution may be determined based on thedistribution 5410. However, it should be understood that ranges ofangular distances may be assigned a same resolution. For example, thehigh resolution zone 5418 may extend from a particular angular distanceon either side of a center of a user's field of view. Optionally,virtual content presented in this high resolution zone 5418 may berendered at a same resolution. Additionally, another example zone (e.g.,a medium resolution zone outside of the high resolution zone 5418) maybe defined by a first angular distance 5412A and a second angulardistance 5412B. Optionally, virtual content presented in this mediumresolution zone may be assigned a same resolution (e.g., a maximumresolution assigned to any angular distance within the portion, anaverage resolution, a lowest resolution, and so on). While FIG. 54illustrates a foveal zone 5406, a high resolution zone 5418, and amedium resolution zone (e.g., between angular distances 5412A, 5412B),it should be understood that the resolution distribution 5410 may besubdivided into multitudes of zones.

The example resolution distribution 5410 may be based, at least in part,on the foveal zone 5406 and a rolloff 5414. As mentioned above withreference to FIGS. 11A1, 11C, and 12A, in some examples, a rolloffattribute of a resolution distribution, such as rolloff 5414 ofresolution distribution 5410, may correspond to a drop-off inresolution. With respect to the example of the distribution 5410 beingGaussian, the rolloff 5414 may be related with a variance and/orstandard deviation. Optionally, the rolloff 5414 may be measured inarcmin per degree of field of view. With respect to a super Gaussiandistribution, the rolloff 5414 may additionally be related with a degreeto which the content of a normal Gaussian's exponent is raised by apower. In embodiments where the distribution 5410 may be governed by amathematical function equivalent or similar to that which describes thefrequency response of a filter, such as a raised-cosine filter, therolloff 5414 may be correspond to a roll-off factor (β) of the function.Thus, the rolloff 5414 may inform how quickly resolution reduces once anangular distance extends outside of the foveal zone 5406.

As will be described below, a wearable display system may learn valuesto utilize for the foveal zone 5406 and rolloff 5414. For example, afirst user may have a foveal zone 5406 which encompasses a greaterangular distance than a second user. As another example, a first usermay have a same angular distance of a foveal zone 5406 as a second user,while the rolloff 5414 may be larger or smaller than the second user's.As another example, the wearable display system may utilize informationaggregated from multitudes of users. For example, the system may utilizea same foveal zone 5406 angular distance for each user. Users may thenadjust the angular distance to improve the functioning of the wearabledisplay system for their unique visual system.

In addition to different users utilizing different foveal zones androlloffs, the wearable display system may customize an angular distanceencompassed by a foveal zone and/or customize a rolloff according to atype of virtual content being presented. For example, virtual contentassociated with a video game may be more perceptible to a user thanvirtual content comprising elements of nature (e.g., virtual treeslocated in their field of view). As an example, the video game contentmay not be similar to real-world content the user has previously seen.That is, the video game may present fantastical situations, characters,and so on, which may be more perceptible to the user than virtual trees,virtual bushes, and so on. As another example, and as will be describedin FIG. 55C, the video game content may have a more varied frequencyspectrum than virtual trees. Thus, in this example optionally thewearable display system may increase angular distances encompassed bythe foveal zone. Optionally, the wearable display system may adjust therolloff as different types of virtual content are presented. Forexample, the rolloff may be larger for the video game than for thenature virtual content. In this example, since the rolloff is larger thesteepness of the distribution 5410 may be reduced. Therefore, the systemmay render virtual content for the video game at a same or greaterresolution than for the nature virtual content.

These values may optionally be utilized to inform sizes, shapes, and soon, of differing resolution adjustment zones (e.g., as described abovein FIGS. 10A-14 ). For example, the foveal zone 5406 may correspond to ahighest resolution of virtual content. Additionally, one or moreadditional zones outside of the foveal zone 5406 may be assigned lesserresolutions at which to present virtual content. FIGS. 55A-55Dillustrate examples of these additional zones along with variations infoveal zone 5406 and rolloff 5414 based on a type of virtual contentpresented.

FIG. 55A illustrates an example scheme to identify a rolloff foraresolution distribution based on a type of virtual content. As describedabove, a resolution distribution may be utilized by a wearable displaysystem to identify a resolution at which to render virtual content basedon its angular distance from a center of a user's field of view. Theresolution distribution may utilize a value of an angular distanceencompassed by a foveal zone, along with a value of a rolloff.

To identify suitable values for the foveal zone and rolloff, users maybe presented with low resolution virtual content via wearable displaysystems in a periphery of their field of view. For example, a wearabledisplay system may select an angular distance at which to presentvirtual content. Optionally, the wearable display system may select anangular distance and a depth at which to present virtual content. Theusers may then indicate whether the virtual content appears blurry orotherwise appears perceptibly reduced in quality. For example, thevirtual content may be presented at a lowered resolution or presentedwith blur applied to it. The periphery may encompass angular distancesoutside of a specified foveal zone. For example, the foveal zone mayencompass a particular range of angular distances from a center of afield of view based on physiological characteristics of eyes. In theexample of FIG. 55A, the foveal zone is specified as encompassing fourdegrees of a user's field of view. That is, the foveal zone has a radiusof two degrees from a center of the user's field of view.

Optionally, and as described above, the display system may updatepresentation of virtual content during a particular eye movement of auser (e.g., a saccade). In this example, a size of a foveal zone may bedependent on a speed of the saccade, and optionally parameters of thedisplay system such as system latency. Latency may include a latencyassociated with obtaining virtual content and updating presentation tothe user. The size of the foveal zone may therefore optionally bedetermined based on the following formula:

${{radial}\mspace{14mu}{size}\mspace{14mu}{of}\mspace{14mu}{foveated}\mspace{14mu}{zone}\mspace{14mu}(\alpha)\mspace{14mu}{FOV}\mspace{11mu}( \deg )} = {{peak}\mspace{14mu}{saccade}\mspace{14mu}{velocity}\mspace{14mu}( \frac{\deg}{\sec} ) \times {total}\mspace{14mu}{system}\mspace{14mu}{latency}\mspace{14mu}( \sec )}$

FIG. 55A illustrates example results 5502 for different participants5508 who were requested to indicate whether presented virtual contentappeared blurry. As described above, the virtual content may bepresented outside of a foveal zone. The virtual content presented to theparticipants 5508 may be separated according to type. Example types 5504may include an action video game, nature, or city scenes. If aparticipant indicated that virtual content appeared blurry, a rolloffassociated with a resolution distribution may be increased by aparticular step size. Once the rolloff is increased, the virtual contentmay be presented to the participant again. That is, the wearable displaysystem may render the virtual content at a new resolution determinedbased on the increased rolloff. Thus, the participant may specify arolloff at which the presented virtual content does not appear blurry.

Based on the responses from the participants, a rolloff 5506 for eachparticipant may be identified. For example, virtual content may bepresented at different angular distances from each participant. Asdescribed in FIG. 54 , different resolutions may be utilized to renderthe virtual content based on angular distance. Each participant may thencause the increase of the rolloff until the participant can't identifythe virtual content as being blurry. A system may then determine alowest rolloff for the participant at which reductions in resolution ofvirtual content were not perceptible. As illustrated in graph 5502, eachparticipant has specified a particular rolloff 5506 according to a type5504 of virtual content presented to the participant. As will bedescribed in more detail below, based on the empirically determinedresult that type 5504 of virtual content may impact rolloff 5502, awearable display system may monitor types of virtual content beingpresented to users. Thus, as the type of virtual content changes, thewearable display system may modify techniques to adjust resolution ofvirtual content.

FIG. 55A further illustrates an example display frustum 5510, along withtwo example resolution adjustment zones 5512, 5514. A first resolutionadjustment zone 5512 is indicated as having a diameter of four degrees.This first resolution adjustment zone 5512 may therefore correspond withthe foveal zone 5406 described above. Based on the determined rolloffs5506 from the aggregated responses of participants 5508, an angulardistance for a second resolution adjustment zone 5514 may be determined.For example, a system may determine a measure of central tendency of therolloffs 5506 (e.g., a mean, a median, and so on). Based on this measureof central tendency, the system may determine an angular distanceextending outside the first resolution adjustment zone 5512 for whichthe resolution is greater than a threshold. With respect to the exampleresolution adjustment distribution 5410 described in FIG. 54 , thethreshold may be greater than the low resolution 5416 illustrated on thedistribution 5410. As another example, the threshold may be greater thana resolution assigned to angular distance 5412A.

In the example of FIG. 55A, the second resolution adjustment zone 5514has been determined to extend ‘8.7’ degrees further than an edge of thefirst resolution adjustment zone 5512. Thus, the determined zones (e.g.,high resolution zone 5118) encompass an angular distance of ‘21.4’degrees of a user's field of view. Optionally, when rendering virtualcontent, a wearable display system may render all virtual content at asame resolution that falls within the second resolution adjustment zone5514. For example, the resolution may be an average of resolutions forthe range of angular distances encompassed by the second resolutionadjustment zone 5514. Optionally, outside of the second resolutionadjustment zone 5514, a wearable display system may render all virtualcontent at a lowest resolution. Optionally, a wearable display systemmay further separate these resolution adjustment zones 5512, 5514. Forexample, the second resolution adjustment zone 5514 may be subdividedinto additional resolution adjustment zones. Each of these subdividedresolution adjustment zones may be assigned a particular resolution atwhich to render virtual content.

While the description above focused on using different participants 5508to identify a rolloff value, it should be understood that the techniquesdescribed above may be customized for each user. For example, a wearabledisplay system may perform a training routine for a user of the wearabledisplay system. The system, as described above, may present virtualcontent at reduced resolutions in the user's periphery. The user maythen indicate when the virtual content is not perceptibly reduced inquality, thus informing a rolloff for the user. The wearable displaysystem may then utilize this rolloff thereafter.

FIG. 55B illustrates an example scheme to identify a rolloff foraresolution distribution based on a type of virtual content. Asillustrated, a foveal zone is indicated as encompassing an angulardistance of ‘8’ degrees. In contrast to FIG. 55A, the foveal zone inthis example is larger. As described above, wearable display systems maypresent reduced resolution virtual content to users in their peripheries(e.g., outside of the foveal zone). The users may then increase a valueof a rolloff, and thus cause virtual content at a same angular distanceto be rendered at a higher resolution. The user may continue increasingthe rolloff until the virtual content is not perceptibly reduced inresolution.

As illustrated, a graph 5512 presents example rolloffs 5512 determinedfor the participants 5508 according to a type 5504 of virtual contentpresented to the participants 5508. Based on the example rolloffs, firstand second resolution adjustment zones 5520, 5522, may be determined.For example, an average rolloff may be determined from the examplerolloffs 5512. Optionally, outliers (e.g., rolloff 5524) may bediscarded. The average rolloff may then be utilized to determine anangular distance of the second resolution adjustment zone 5522. Anangular distance encompassed by the first resolution adjustment zone5520, for example a foveal zone, is eight degrees as described above.Based on this angular distance, and the determined rolloffs 5516, anangular distance for the second resolution adjustment zone 5522 isindicated as extending ‘5.5’ degrees from an edge of the first zone5520. The first and second zones 5520, 5522, therefore encompass 19degrees of a user's field of view. Thus, the second resolutionadjustment zone 5522 encompasses a smaller angular distance than thesecond resolution adjustment zone 5514 of FIG. 55B.

FIG. 55C illustrates a graph 5530 of average rolloffs determined fordifferent types of image content. FIG. 55C illustrates rolloffs fordiffering angular distances encompassed by foveal zones. For example,the first foveal zone 5532 of FIG. 55A encompasses four degrees of auser's field of view. As another example, the second foveal zone 5534 ofFIG. 55B encompasses eight degrees of a user's field of view. Thesedifferent foveal zones 5532, 5534, may result in different determinedaverage rolloffs. As illustrated, the first foveal zone 5532 may requiregreater average rolloffs, and thus a greater angular distanceencompassed by a second resolution adjustment zone, than the secondfoveal zone 5534.

As described above, the average rolloffs are dependent on a type ofimage content being presented to the participants. Indeed, rolloffs maybe determined to be greater for image content with varied frequencydependence. That is, virtual content illustrating nature or calm cityscenes may have an inverse spatial frequency spectrum (e.g., a powerspectral density is inversely proportional to frequency) to a flatspatial frequency spectrum. In contrast, virtual content illustratingvideo game action may have a more varied frequency spectrum. Thus,virtual content illustrating synthetic images may tend to require moregradual rolloffs as compared to virtual content illustrating naturalimages.

FIG. 55D illustrates a graph 5540 of average rolloffs determined fordifferent types of presented image noise. The types of noise presentedinclude pink noise 5542 and white noise 5544. As is known in the art,white noise includes signals with a power spectral density that issubstantially flat in frequency. Pink noise includes signals with apower spectral density that is inversely proportional to frequency.Thus, with respect to pink noise intensities of higher frequencies arereduced as compared to white noise. Since pink noise reduces higherfrequencies in intensity, it may be understood that virtual contentconforming to pink noise may appear less chaotic or random to users.Thus, users may be less able to identify whether the pink noise wasblurred when presented in their periphery. In contrast, white noise maybe more noticeable (e.g., the white noise may appear sharper to users).Thus, users may be more able to identify whether the white noise wasreduced in resolution when presented in their periphery.

As illustrated in FIG. 55D, average rolloffs were determined for thediffering angular distances of foveal zones described above (e.g., 4degrees 5546 and 8 degrees 5548). In conformance with the theorydescribed herein, average rolloffs for white noise 5544 were determinedto be greater than average rolloffs for pink noise 5542. That is, whitenoise is more similar to the video game virtual content than the naturevirtual content described in FIGS. 55A-55C. Similarly, an angulardistance encompassed by the foveal zone and second resolution adjustmentzones (e.g., as illustrated in FIGS. 55A-55B), which in the illustrationis referred to as a ‘high resolution tunnel’, is greater for the whitenoise 5544 than for the pink noise 5542. That is, since the rolloffsdetermined for the white noise 5544 are greater, a width of acorresponding resolution distribution (e.g., as illustrated indistribution 5410 in FIG. 54 ) may also be greater. Thus, the resolutiondistribution for white noise 5544 encompasses a larger angular distancefor which resolutions are greater than a threshold (e.g., greater than alow resolution 5416).

Thus, it may be appreciated that a resolution distribution may be basedon features of a foveal zone and a rolloff. As an example, a foveal zonemay be defined based on an angular distance encompassed by the fovealzone. For example, and with respect to FIG. 55B, an example angulardistance may be eight (8) degrees. With respect to the resolutiondistribution being a Gaussian or raised-cosine distribution, the fovealzone may thus correspond with a plateau portion (e.g., plateau 5406 inFIG. 54 ). As described above, the rolloff may additionally impact aresolution distribution. For example, the rolloff may more sharplyreduce a resolution as a virtual object is located farther from a fovealzone.

In some embodiments, certain features of a resolution distribution maybe adjusted during operation. For example, a rolloff may be adjusted. Inthis example, the rolloff may, as an example, be adjusted based on auser preference (e.g., the user may notice a reduced resolution and, asa result, specify a more gradual rolloff with distance from the fixationpoint). As another example, a plateau width (e.g., angular distanceencompassed by a foveal zone) may be adjusted. In some embodiments,certain features of a resolution distribution may be held constantduring operation. For example, an area under a resolution distributionmay be held constant. In this example, the average value (e.g., mean ofthe distribution) may be held constant, such that an average resolutionmay be achieved. Thus, if a rolloff is adjusted then the plateau widthmay be adjusted. Similarly, if the plateau width is adjusted then therolloff may be adjusted. As an example, the display system maydynamically redistribute pixels (e.g., such that the resolution in agiven region of the user's field of view may be dynamically increasedand decreased), but it may not actually adjust the total quantity ofpixels available to represent virtual content.

In some embodiments, certain features of a resolution distribution mayhave constraints. For example, an angular distance encompassed by afoveal zone may have a minimum (e.g., a minimum plateau width). Asanother example, the resolution distribution may be constrained suchthat it never falls below a specific minimum width (e.g., the angularwidth of an average human fovea or the radial size of foveated zone).

Example Flowcharts

FIG. 56 illustrates a flowchart of an example process 5600 fordetermining a rolloff to be utilized in a resolution distribution. Forconvenience, the process 5600 may be described as being performed by adisplay system (e.g., the wearable display system 60, which may includeprocessing hardware and software, and optionally may provide informationto an outside system of one or more computers or other processing, forinstance to offload processing to the outside system, and receiveinformation from the outside system).

The process 5600 describes a user training the display system to theuser's particular visual acuity. The user may view virtual content in aperiphery of the user's field of view (e.g., outside of a foveal zone),and may specify whether the virtual content appears blurry. If the userpositively specifies the virtual content as being blurry, the displaysystem may increase a rolloff associated with a resolution distribution.The same, or different, virtual content may then be presented to theuser. Since the rolloff was increased, the virtual content may berendered at a greater resolution. The user may then indicate whether therolloff still appears blurry. In this way, the display system maydetermine a rolloff for the user such that reductions in resolution areimperceptible. Additionally, and as described above in FIGS. 55A-55D,the process 5600 may be performed for multitudes of users. Based onresponses of these users, an average rolloff may be determined. Thisaverage rolloff may then be used for all users. For example, the averagerolloff may be used as a default rolloff. A user may then adjust thisaverage rolloff according to the techniques described herein.

At block 5602, the display system access information identifying aresolution distribution. To identify a resolution at which to rendervirtual content located in a user's field of view, the display systemmay utilize a resolution distribution (e.g., resolution distribution5410 illustrated in FIG. 54 ). As described above, the resolutiondistribution may inform selection of a resolution based on an angulardistance of virtual content from a center of a user's field of view. Aswill be described below, to inform a shape associated with theresolution distribution, the display system may adjust a rolloff (e.g.,a Gaussian rolloff) utilized in the resolution distribution.

At block 5604, the display system obtains virtual content of aparticular type (block 5604). Since, as described above with respect toFIGS. 55A-55D, users may have a greater ability to notice reductions inresolution of certain types of virtual content, the display system mayadvantageously present differing types of virtual content. For example,the types may include video game virtual content, nature-based virtualcontent, office-based virtual content (e.g., documents, spreadsheets,animations, and so on), calm city scenes, suburban scenes (e.g., trees,houses), and so on. At block 5604, the display system may thereforeselect from among these differing types. For example, the display systemmay select a particular type of virtual content and then keep presentingvirtual content of the selected type until a user of the display systemceases adjusting a rolloff.

At block 5606, the display system renders the obtained virtual contentoutside of a foveal zone. The display system may identify a resolution(e.g., reduced from a highest resolution) at which to render theobtained virtual content based on the accessed resolution distribution.As described above, with respect to at least FIG. 12A, the displaysystem may determine a point at which the user is fixating. Thisfixation point (e.g., three-dimensional fixation point) may be utilizedas a center of the user's field of view. As describe above, theresolution distribution may be defined, at least in part, by a value ofan angular distance encompassed by a foveal zone and a value of arolloff. The display system may thus select an angular distance, fromthe determined center of the user's field of view, at which to renderthe obtained virtual content.

For example, and with respect to FIG. 54 , the display system mayoptionally utilize an angular distance greater than an edge of a fovealzone 5406, but less than an edge corresponding to a low or lowestresolution 5416. For example, the angular distance may be included in ahigh resolution zone, medium resolution zone, and so on as describedabove. The display system may then obtain, based on the selected angulardistance and resolution distribution, a resolution at which to renderthe virtual content. Optionally, and as described above, the resolutiondistribution may be divided into zones. In this example, the displaysystem may utilize a greatest resolution within a zone that includes theselected angular distance. As another example, the display system mayutilize an average resolution within the zone.

Furthermore, the resolution distribution may depend on a depth along theangular distance. That is, and as described above in FIGS. 11A1-11E, aresolution at which to render virtual content may depend onthree-dimensional distance of the virtual content from a user's fixationpoint. While the description above has focused on angular distance, itshould be understood that the resolution distribution accessed in block5602 may include depth information. Thus, the resolution distributionmay be, for example, a multivariate normal distribution. In thisexample, the display system may select an angular distance andoptionally a depth from the user. As described in FIGS. 11A1-11E,increased depth may cause a reduction in resolution. However, forangular distances that are included in a foveal zone, virtual content atany depth along the angular distances may be rendered at a highresolution.

The display system may then present the rendered virtual content to theuser of the display system. For example, the virtual content may bepresented at the selected angular distance (e.g., a centroid of thevirtual content may correspond to a three-dimensional location along theselected angular distance). As another example, the virtual content maybe presented at a particular depth from the user as described above.

At block 5608, the display system receives a response indicating whetherthe user may detect if the virtual content has been reduced inresolution. For example, the display system may respond to user inputobtained from one or more devices (e.g., controllers, remotes, and soon). As another example, the display system may monitor movements of auser's hands or other extremities. In this example, the display systemmay determine that the user is performing a particular hand motion toindicate that blurriness is evident (e.g., the user may wave his/herhands back and forth). As another example, a shaking of the user's headfrom left to right may indicate a response. For example, the responsemay indicate a ‘no’, such that the user cannot detect the blurriness.Similarly, a shaking of the user's head up and down may indicate a‘yes’, such that the user may detect the blurriness.

If the user may detect that the virtual content has been reduced inresolution, at block 5610 the display system may increase the rolloff.For example, the rolloff may be increased by a particular step size,such as 0.3, 0.6, 0.7, or 1.1 arcmin/deg. The display system may thenrender the same, or different, virtual content based on the increasedrolloff. Blocks 5606 through 5610 may be repeated until the userindicates he/she cannot identify a reduction in resolution.

If the user cannot detect that the virtual content has been reduced inresolution, at block 5612 the display system may store the rolloff.Optionally, the stored rolloff may be utilized for the user thereafter.Since rolloffs, as described above, may be dependent on a type ofvirtual content, the display system may determine rolloffs for thediffering types. When presenting virtual content of a certain type, thedisplay system may optionally utilize a rolloff specific to the type.

FIG. 57 illustrates an example flowchart of a process 5700 forpresenting virtual content according to a type of the virtual content.For convenience, the process 5700 may be described as being performed bya display system (e.g., the wearable display system 60, which mayinclude processing hardware and software, and optionally may provideinformation to an outside system of one or more computers or otherprocessing, for instance to offload processing to the outside system,and receive information from the outside system).

At block 5702, the display system determines a fixation point of a user.For example, the fixation point may be a three-dimensional fixationpoint. As described above, with respect to at least FIG. 12A, thedisplay system may monitor eyes of the user and identify a location atwhich vectors extending from each eye intersect (e.g., a vergencepoint).

At block 5704, the display system obtains location informationassociated with virtual content to be presented. The display system mayidentify a location, such as a three-dimensional location, at whichparticular virtual content is to be presented. As described in FIGS.54-57 , the location may optionally be specified according to polarcoordinates (e.g., an angle from a center of the user's field of view,and a distance along the angle).

At block 5706, the display system identifies, or obtains, aresolution-modifying parameter. An example resolution-modifyingparameter may be a type of the virtual content. Another exampleresolution-modifying parameter may include a user preference. Forexample, the user preference may indicate an adjustment to a resolutiondistribution (e.g., as described in FIG. 56 ). With respect to theexample of a type that particular content may be categorized as, thedisplay system may optionally access metadata, or other information,that indicates the type. In this example, if the user is playing a videogame, the display system may access information specifying the videogame. As another example, the display system may analyze a powerspectral density of the virtual content. The display system may thenidentify whether the virtual content is more similar to a particulartype of virtual content (e.g., video game data, nature, and so on asdescribed herein). In this example, the display system may determinethat the frequency spectrum is varied, or that the frequency spectrum iscloser to a particular type of noise (e.g., pink noise, white noise).Based on these comparisons, the display system may select a closest typeof virtual content to the obtained virtual content.

At block 5708, the display system identifies a resolution at which torender the virtual content. As described in FIG. 54 , the display systemmay utilize a resolution distribution to identify the resolution. Basedon the obtained virtual content's location information, and identifiedtype of virtual content, the display system may prefer to utilize aparticular resolution distribution. For example, the particularresolution distribution may have a rolloff determined for the identifiedtype of virtual content (e.g., as described in FIG. 56 ). Optionally,for a type of virtual content which is similar to two or more types ofvirtual content, the display system may combine characteristics ofresolution distributions specific to each. For example, the displaysystem may utilize an average value of a size of a foveal zone or anaverage value of a rolloff. As another example, the system may utilize alargest value of a rolloff to ensure virtual content is rendered at ahighest resolution.

At block 5710, the display system renders the virtual content. Thedisplay system may thus render the virtual content at the identifiedresolution. As described above, the user may update the resolutiondistribution if the user identifies blurriness in presented virtualcontent. For example, the user may update a rolloff via settings of thewearable display system. As another example, the user may classifyvirtual content as corresponding to particular types. In this way, ifthe display system incorrectly identifies the type of virtual content inblock 5706, the user may update the classification.

Adjusting Resolution of Virtual Content

Virtual content may be adjusted in resolution to reduce processing andpower requirements. Examples of adjusting resolution may include, forexample, adjusting a polygon count, texture information, shader orlighting effects, and so on. As described above with respect to at leastFIGS. 11A-11E, different resolution adjustment zones may be utilized.Virtual content located within each zone may be adjusted based on aresolution assigned, or determined, for that zone. As described above, aparticular resolution adjustment zone (e.g., a foveal zone) mayencompass a particular angular distance within a user's field of view,and virtual content within this particular resolution adjustment zonemay be rendered at a highest resolution.

A portion of a user's field of view, referred to herein as a highresolution zone or tunnel (e.g., high resolution zone 5418 illustratedin FIG. 54 ), may encompass the foveal zone and one or more zonesextending outwards from the foveal zone. For virtual content presentedin this high resolution zone, the display system may identify aresolution at which to render the virtual content. As illustrated inFIGS. 55A-55D, an example angular distance of a high resolution zone mayencompass between about 18 degrees to about 20 degrees of a user's fieldof view. This example angular distance may optionally be increased dueto one or more sources of error. Additionally, this example angulardistance may include a notch in it to account for a blind spot of auser. Virtual content located in this notch may be advantageously bereduced in resolution.

For virtual content located proximate to an edge of a resolutionadjustment zone, or for virtual content that encompasses an edge, thedisplay system may adjust a resolution distribution, size, position,and/or geometry associated with the resolution adjustment zones.Optionally, if the virtual content is of relatively stark contrast, thedisplay system may adjust the parameters described above. This isbecause the boundary between resolution adjustment zones (e.g., a fovealzone and surrounding zones) may become more discernible to a user in thepresence of such an edge or border of stark contrast.

In addition, adjusting resolution may include the display systemadjusting a refresh rate associated with different portions of a user'sfield of view. For example, as described above with respect to at leastFIG. 43 , a display system 21000 may include a foveal tracker 21006,which may take the form of a scanning mirror (e.g., MEMs mirror). Inthis example, the display system 21000 may utilize at least twomultiplexed virtual content images (e.g., multiplexed in time orpolarization) to present both high resolution and low resolution virtualcontent to a user. These different resolutions may be generated via asame MEMS mirror (e.g., as described above). As described in FIGS.38A-38B, a high-resolution image with low field of view (e.g., imagestream 16020) may be positioned within a central vision of a user andmay correspond to a native MEMS project field. A low-resolution imagethat covers a high field of view, for example image stream 16010), maybe an optically expanded version of the MEMS projector field. The highresolution low field of view region may be tracked, for example based onan eye tracker. This may enable a lower scan angle and speed requirementfor the MEMS mirror.

As will be described below, the display system may optionally apply blurto virtual content located proximate to edges of these resolutionadjustment zones. FIGS. 58A-59 illustrate examples of blurring regionsbetween resolutions adjustment zones. These blurring regions may beutilized to hide a boundary between a higher resolution adjustment zoneand a lower resolution adjustment zone.

FIG. 58A illustrates two example blurring regions 5802, 5808. Theblurring regions may cause virtual content located, at least in part,within the blurring regions to be blurred, thus masking the transitionbetween a low-resolution zone (e.g., zone 5806) and a higher resolutionzone (e.g., zone 5804). For example, virtual content located partiallyin the low-resolution zone 5806, and partially in the blurring region5802, may be rendered at a resolution corresponding to thelow-resolution zone 5806. The blurring may therefore limit an extent towhich a user is aware of the reduced resolution. Optionally, the virtualcontent may be rendered according to the high-resolution zone 5808, anda portion extending into the blurring region 5806 may be blurred. Theblurring regions 5802, 5808, may be of a particular size and/or shape.For example, the blurring regions may form a starburst pattern. Thisstarburst pattern may optionally be extended in depth according to thetechniques described in FIGS. 11A-11E.

As described above in FIG. 14 , an example blurring process may includethe display system performing a convolution of a kernel associated withblurring (e.g., a Gaussian kernel, circular kernel such as to reproducea bokeh effect, box blur, and so on) to the content. In this way, thereduction in resolution may be masked, while the processing savings fromreducing the resolution may be maintained. Optionally, a strengthassociated with the blurring process (e.g., a degree to which thecontent is blurred) may be based on a difference in depth between theuser's fixation point and the content, and/or an angular proximity ofthe content to the user's gaze. For example, the degree of blurring mayincrease with increasing proximity to the user's gaze.

FIG. 58B illustrates two additional example blurring regions 5814-5818.These blurring regions 5814-5818, may be similar to the blurring regionsof FIG. 58A, but may be different in one or more of shape, size, and soon. For example, the starburst pattern of blurring region 5814 may belarger than the starburst pattern of blurring regions 5802, 5808.Additionally, for portion 5820, two blurring regions are illustrated.For example, blurring region 5816 may separate a zone of low resolutionfrom a zone of medium resolution. Additionally, blurring region 5818 mayseparate a zone of medium resolution from a zone of high resolution.

FIG. 59 illustrates an example 5900 of different resolution adjustmentzones according to the techniques described herein. In the illustration,a first zone 5902 may correspond to a foveal zone. As described above,the foveal zone may encompass a user's fixation point 5903.Additionally, a second zone 5904 (e.g., outside of the first zone 5902)is illustrated along with a third zone 5906. Each of these resolutionadjustment zones may cause resolutions of virtual content located withinthe zone to be rendered at a particular resolution. Additionally, anedge between zones (e.g., edge 5908) may be associated with a blurringregion as illustrated in FIGS. 58A-58B.

The various aspects, embodiments, implementations or features of thedescribed embodiments can be used separately or in any combination.Various aspects of the described embodiments can be implemented bysoftware, hardware or a combination of hardware and software. Thedescribed embodiments can also be embodied as computer readable code ona computer readable medium for controlling manufacturing operations oras computer readable code on a computer readable medium for controllinga manufacturing line. The computer readable medium is any data storagedevice that can store data, which can thereafter be read by a computersystem. Examples of the computer readable medium include read-onlymemory, random-access memory, CD-ROMs, HDDs, DVDs, magnetic tape, andoptical data storage devices. The computer readable medium can also bedistributed over network-coupled computer systems so that the computerreadable code is stored and executed in a distributed fashion.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the describedembodiments. However, it will be apparent to one skilled in the art thatthe specific details are not required in order to practice the describedembodiments. Thus, the foregoing descriptions of specific embodimentsare presented for purposes of illustration and description. They are notintended to be exhaustive or to limit the described embodiments to theprecise forms disclosed. It will be apparent to one of ordinary skill inthe art that many modifications and variations are possible in view ofthe above teachings.

It will also be appreciated that each of the processes, methods, andalgorithms described herein and/or depicted in the figures may beembodied in, and fully or partially automated by, code modules executedby one or more physical computing systems, hardware computer processors,application-specific circuitry, and/or electronic hardware configured toexecute specific and particular computer instructions. For example,computing systems may include general purpose computers (e.g., servers)programmed with specific computer instructions or special purposecomputers, special purpose circuitry, and so forth. A code module may becompiled and linked into an executable program, installed in a dynamiclink library, or may be written in an interpreted programming language.In some embodiments, particular operations and methods may be performedby circuitry that is specific to a given function.

Further, certain embodiments of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. In some embodiments,the non-transitory computer-readable medium may be part of one or moreof the local processing and data module (140), the remote processingmodule (150), and remote data repository (160). The methods and modules(or data) may also be transmitted as generated data signals (e.g., aspart of a carrier wave or other analog or digital propagated signal) ona variety of computer-readable transmission mediums, includingwireless-based and wired/cable-based mediums, and may take a variety offorms (e.g., as part of a single or multiplexed analog signal, or asmultiple discrete digital packets or frames). The results of thedisclosed processes or process steps may be stored, persistently orotherwise, in any type of non-transitory, tangible computer storage ormay be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities may be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto may be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe embodiments described herein is for illustrative purposes and shouldnot be understood as requiring such separation in all embodiments. Itshould be understood that the described program components, methods, andsystems may generally be integrated together in a single computerproduct or packaged into multiple computer products.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than restrictive sense.

Indeed, it will be appreciated that the systems and methods of thedisclosure each have several innovative aspects, no single one of whichis solely responsible or required for the desirable attributes disclosedherein. The various features and processes described above may be usedindependently of one another, or may be combined in various ways. Allpossible combinations and subcombinations are intended to fall withinthe scope of this disclosure.

Certain features that are described in this specification in the contextof separate embodiments also may be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment also may be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

It will be appreciated that conditional language used herein, such as,among others, “can,” “could,” “might,” “may,” “e.g.,” and the like,unless specifically stated otherwise, or otherwise understood within thecontext as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or withoutauthor input or prompting, whether these features, elements and/or stepsare included or are to be performed in any particular embodiment. Theterms “comprising,” “including,” “having,” and the like are synonymousand are used inclusively, in an open-ended fashion, and do not excludeadditional elements, features, acts, operations, and so forth. Also, theterm “or” is used in its inclusive sense (and not in its exclusivesense) so that when used, for example, to connect a list of elements,the term “or” means one, some, or all of the elements in the list. Inaddition, the articles “a,” “an,” and “the” as used in this applicationand the appended claims are to be construed to mean “one or more” or “atleast one” unless specified otherwise. Similarly, while operations maybe depicted in the drawings in a particular order, it is to berecognized that such operations need not be performed in the particularorder shown or in sequential order, or that all illustrated operationsbe performed, to achieve desirable results. Further, the drawings mayschematically depict one more example processes in the form of aflowchart. However, other operations that are not depicted may beincorporated in the example methods and processes that are schematicallyillustrated. For example, one or more additional operations may beperformed before, after, simultaneously, or between any of theillustrated operations. Additionally, the operations may be rearrangedor reordered in other embodiments. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system components in the embodiments describedabove should not be understood as requiring such separation in allembodiments, and it should be understood that the described programcomponents and systems may generally be integrated together in a singlesoftware product or packaged into multiple software products.Additionally, other embodiments are within the scope of the followingclaims. In some cases, the actions recited in the claims may beperformed in a different order and still achieve desirable results.

Accordingly, the claims are not intended to be limited to theembodiments shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

What is claimed is:
 1. A head-mounted display system comprising: adisplay configured to output light with a plurality of levels ofwavefront divergence, wherein different levels of wavefront divergencecorrespond to different distances from a fixation point of a user'seyes; one or more processors; and one or more computer storage mediastoring instructions that when executed by the one or more processors,cause the one or more processors to perform operations comprising:determining the fixation point of the user's eyes; obtaining locationinformation associated with a first virtual object to be presented tothe user via the display; obtaining a resolution-modifying parameter ofthe first virtual object; identifying, based on the location informationand the resolution-modifying parameter of the first virtual object, aparticular resolution at which to render the first virtual object,wherein the particular resolution is based on a resolution distributionspecifying resolutions for corresponding distances from the fixationpoint, wherein identifying the particular resolution comprisesassociating different resolutions with different levels of wavefrontdivergence, wherein a field of view of the user is separated into aplurality of portions based on the resolution distribution, and wherein,for a particular depth or range of depths, each portion is assigned anassociated resolution at which to render virtual content; and causingpresentation to the user, via the display, of the first virtual objectrendered at the identified resolution by outputting light from thedisplay with a level of wavefront divergence associated with theidentified resolution.
 2. The head-mounted display system of claim 1,wherein the resolution-modifying parameter comprises a content typeassociated with the first virtual object, wherein the operations furthercomprise: accessing a plurality of resolution distributions, theplurality of resolution distributions associated with respective contenttypes; and selecting, from the plurality of resolution distributions, aparticular resolution distribution based on the content type of thefirst virtual object, wherein the particular resolution comprises theparticular resolution distribution.
 3. The head-mounted display systemof claim 2, wherein the content type associated with the first virtualobject is identified based on a frequency spectrum associated with thefirst virtual object.
 4. The head-mounted display system of claim 3,wherein the plurality of resolution distributions are associated withrespective rolloffs in resolution away from the fixation point, whereinvalues of the rolloffs differ for content with different frequencyspectrums.
 5. The head-mounted display system of claim 1, wherein theresolution-modifying parameter is a user selectable value.
 6. Thehead-mounted display system of claim 5, wherein the display isconfigured to adjust the particular resolution, and wherein adjustingthe particular resolution comprises: causing presentation to the user,via the display, of a second virtual object, the second virtual objectbeing rendered with the resolution distribution identified for the firstvirtual object; receiving, from the user, a response indicating userdetection of reduction in resolution of the second virtual object,wherein the user response is the user selectable value; and adjustingthe particular resolution distribution.
 7. The head-mounted displaysystem of claim 6, wherein adjusting the particular resolutiondistribution comprises: adjusting a rolloff associated with theparticular resolution distribution, wherein adjusting a rolloff changesan amount of resolution reduction based on a distance from the center ofthe field of view of the user.
 8. The head-mounted display system ofclaim 1, wherein the fixation point is in a volume at the center of thefield of view of the user.
 9. The head-mounted display system of claim1, wherein the operations further comprise: determining a proximity ofthe first virtual object to a boundary of one of the plurality ofportions; and modifying a presentation of the first virtual objectedbased on the determined proximity.
 10. The head-mounted display systemof claim 1, wherein modifying a presentation of the first virtualobjected based on the determined proximity comprises applying a blurringprocess to the virtual object.
 11. The head-mounted display system ofclaim 1, wherein identifying the particular resolution at which torender the first virtual object comprises: identifying a portion of theplurality of portions which encompasses the first virtual object; andidentifying the resolution based on the portion which encompasses thefirst virtual object.
 12. A computer-implemented method, the methodbeing performed by a head-mounted display system comprising one or moreprocessors, and the method comprising: determining a fixation point of auser's eyes; obtaining location information associated with a firstvirtual object to be presented to the user via a display configured tooutput light with a plurality of levels of wavefront divergence, whereindifferent levels of wavefront divergence correspond to differentdistances from the fixation point of the user's eyes; obtaining aresolution-modifying parameter of the first virtual object; identifying,based on the location information and the resolution-modifying parameterof the first virtual object, a particular resolution at which to renderthe first virtual object, wherein the particular resolution is based ona resolution distribution specifying resolutions for correspondingdistances from the fixation point, wherein identifying the particularresolution comprises associating different resolutions with differentlevels of wavefront divergence, wherein a field of view of the user isseparated into a plurality of portions based on the resolutiondistribution, and wherein, for a particular depth or range of depths,each portion is assigned an associated resolution at which to rendervirtual content; and causing presentation to the user, via the display,of the first virtual object rendered at the identified resolution byoutputting light from the display with a level of wavefront divergenceassociated with the identified resolution.
 13. The computer-implementedmethod of claim 12, wherein the resolution-modifying parameter comprisesa content type associated with the first virtual object, wherein themethod further comprises: accessing a plurality of resolutiondistributions, the plurality of resolution distributions associated withrespective content types; and selecting, from the plurality ofresolution distributions, a particular resolution distribution based onthe content type of the first virtual object, wherein the particularresolution comprises the particular resolution distribution.
 14. Thecomputer-implemented method of claim 13, wherein the content typeassociated with the first virtual object is identified based on afrequency spectrum associated with the first virtual object.
 15. Thecomputer-implemented method of claim 13, wherein the plurality ofresolution distributions are associated with respective rolloffs inresolution away from the fixation point, wherein values of the rolloffsdiffer for content with different frequency spectrums.
 16. Thecomputer-implemented method of claim 13, wherein theresolution-modifying parameter is a user selectable value. 17.Non-transitory computer storage media storing instructions that whenexecuted by a head-mounted display system of one or more processors,cause the one or more processors to perform operations comprising:determining a fixation point of a user's eyes; obtaining locationinformation associated with a first virtual object to be presented tothe user via a display configured to output light with a plurality oflevels of wavefront divergence, wherein different levels of wavefrontdivergence correspond to different distances from the fixation point ofthe user's eyes; obtaining a resolution-modifying parameter of the firstvirtual object; identifying, based on the location information and theresolution-modifying parameter of the first virtual object, a particularresolution at which to render the first virtual object, wherein theparticular resolution is based on a resolution distribution specifyingresolutions for corresponding distances from the fixation point, whereinidentifying the particular resolution comprises associating differentresolutions with different levels of wavefront divergence, wherein afield of view of the user is separated into a plurality of portionsbased on the resolution distribution, and wherein, for a particulardepth or range of depths, each portion is assigned an associatedresolution at which to render virtual content; and causing presentationto the user, via the display, of the first virtual object rendered atthe identified resolution by outputting light from the display with alevel of wavefront divergence associated with the identified resolution.18. The non-transitory computer storage media of claim 17, wherein theresolution-modifying parameter comprises a content type associated withthe first virtual object, wherein the operations further comprise:accessing a plurality of resolution distributions, the plurality ofresolution distributions associated with respective content types; andselecting, from the plurality of resolution distributions, a particularresolution distribution based on the content type of the first virtualobject, wherein the particular resolution comprises the particularresolution distribution.
 19. The non-transitory computer storage mediaof claim 18, wherein the content type associated with the first virtualobject is identified based on a frequency spectrum associated with thefirst virtual object.
 20. The non-transitory computer storage media ofclaim 18, wherein the plurality of resolution distributions areassociated with respective rolloffs in resolution away from the fixationpoint, wherein values of the rolloffs differ for content with differentfrequency spectrums.